Skip to content

CHANGELOG.md

Latest commit

 

History

History
131 lines (118 loc) · 36.4 KB

CHANGELOG.md

File metadata and controls

131 lines (118 loc) · 36.4 KB
Raw

Changelog

All notable changes to this project will be documented in this file. The format follows Keep a Changelog, and versioned releases follow Semantic Versioning.

[Unreleased]

Phase 0 and the Phase 1 MVP vertical slice are complete. The Phase 2 reliable-execution foundation and a trusted-local MMLU-Pro/GPQA-Diamond evaluation slice are implemented, while Phase 2 and Phase 3 as a whole remain in_progress. This is a development baseline, not a published release or production/HA claim.

Added

  • Added the P2-07 recovery-operations kickoff package: ADR-0016, an independent execution plan, and a work log now freeze a deliberately narrow PostgreSQL 16 dump/restore verification, separate keyring pairing, Redis-as-notification rebuild, alert-drill, destructive-operation, and evidence boundary. This commit contains documentation only; P2-07 implementation and qualification have not started, Phase 2 remains in_progress, and no production DR/PITR/RPO/RTO/HA claim is made.
  • Added the completed P2-06 implementation under ADR-0015: GET /api/v1/metrics/prometheus renders a fixed Prometheus text 0.0.4 gauge set from one DB-time snapshot, with enum-only labels, bounded 15-minute audit and one-hour latency windows, fail-closed audit validation, and one in-flight collection per API process. Repository-owned configuration supplies exactly eight alert rules and fixed Operations runbook links; Prometheus, Alertmanager, and notification delivery are not deployed by this repository.
  • Added Alembic revision 20260828_0005, the worker_processes generation table, bounded (expires_at,id) / (occurred_at,id) audit scan indexes, and DB-time coalesced Worker scan/claim/lease-heartbeat/progress recording. /tasks/metrics and the exporter expose only low-cardinality aggregates; the dependency probe remains capability-only. The stopped SQLite-to-PostgreSQL importer now copies and reconciles all 13 application tables, rejects a still-live Worker generation, and preserves stopped/stale process facts.
  • Added llmbenchlab-audit-retention archive|verify|reconcile|restore|delete. The v1 canonical JSONL archive is bounded, exact-digest bound, atomically installed without replacement at 0600, and independently verifiable without a database engine; delete is never implicit, restore/delete require an exact confirmed archive SHA-256, and commit-unknown outcomes require read-only reconciliation before retry. These integrity hashes are neither signatures nor WORM guarantees.
  • Added the Phase 2 single-host control-plane qualification under ADR-0012 through ADR-0014. The P2-local-control-plane-v2 harness fixes one warm-up plus exactly five measured trials, four Mock-only capacity cells, paired Worker scaling, queue/execution/end-to-end and recovery objectives, ledger-derived correctness gates, bounded capacity formulas, and scoped project-image cleanup. Clean implementation SHA b6a35fef1dd069ebb54b69955058915c722aa34d passed 23/23 SLO checks with all six child invariants and cleanup checks; the earlier v1 aggregate remains permanently unqualified rather than being discarded or reclassified. This qualifies only the recorded single-host Mock control-plane profile, not production SLA, real-Provider capacity, HA, billing, or external exactly-once behavior.
  • Added the current Phase 2 governance/audit slice under ADR-0009, as amended by ADR-0010 and ADR-0011: database-authoritative global/provider/model/run admission, immutable policy/hash plus per-Run override snapshots, per-Provider-attempt reservation/settlement ledger, fixed-window RPM/TPM and lifetime request/Token/cost limits, bounded backlog, cooperative question quantum, typed audit/history APIs, and non-secret Provider evidence.
  • Added Alembic revision 20260827_0004 with six governance/audit tables (governance_policies, governance_scopes, governance_minute_buckets, question_executions, provider_call_reservations, and audit_events) plus Run/Response operational evidence. The stopped SQLite-to-PostgreSQL importer now copies and reconciles all 12 application tables in dependency order.
  • Added Mock-only Phase 2 capacity tooling for finite policy read-back, one/two-Worker measurements, concurrent backlog 202/429 admission, sub-Run question quantum, cross-Model fairness, Worker loss, Redis interruption, and ledger/audit reconciliation. Real PostgreSQL integration coverage now includes four-scope concurrency, RPM/TPM and lifetime budgets, concurrent backlog, settlement/reconciliation races, and idempotent audit replay.
  • Extended the Phase 2 acceptance harness with three explicitly named deterministic database-seam injections: reserved before send-start, send_started before settlement, and Response committed before final local recovery. They validate pre-send release/retry preservation, conservative settlement, and exactly-once local Response/ledger/audit evidence without pretending that a process SIGKILL can be timed to a sub-millisecond seam.
  • Phase 0 governance, architecture decisions, protocols, roadmap, and repeatable work-log process.
  • FastAPI, SQLAlchemy, Alembic, SQLite, and Mock-adapter backend for the reproducible MVP evaluation flow.
  • React and TypeScript interface for models, Benchmarks, runs, result evidence, summary metrics, and the leaderboard.
  • Offline unit, integration, component, and vertical-slice smoke tests that require no provider credential.
  • Migration regression tests for clean databases, legacy-data preservation, consistent SQLite backups, schema-drift rejection, idempotency, and startup revision checks.
  • Local setup and development scripts, unified Make targets, and an optional two-service Docker Compose deployment with persistent SQLite storage.
  • GitHub Actions checks for backend lint/tests and frontend lint/tests/build, plus issue and pull request templates.
  • MIT License and contribution guidance for the initial open-source repository.
  • Alembic revision 20260825_0002, PostgreSQL deployment support, database-clock Run leases, monotonic fencing tokens, heartbeats, bounded retry/dead-letter metadata, and idempotent per-question persistence.
  • Redis Streams at-least-once notifications plus an independent Worker that reconciles the database, claims leases, resumes missing Responses, and acknowledges messages only after database disposition.
  • /live, DB-only /health, dependency-aware /ready, database-derived task gauges, request/Run correlation, and sanitized JSON for LLMBenchLab application loggers.
  • A six-service local Compose topology (postgres, redis, one-shot migrate, api, worker, and frontend) and CI jobs for SQLite, real PostgreSQL/Redis integration, and full-stack fault acceptance.
  • An explicit stopped-SQLite to offline-empty-PostgreSQL importer with read-only source validation, transactional locking/copy, content-free reconciliation digests, and distinct rollback/commit-uncertainty/post-commit-verification outcomes.
  • Public organization repository at CWNU-Open-Source-Community/LLMBenchLab, a visible CI badge, and a repository-wide stage gate requiring each commit to be pushed and its exact GitHub Actions SHA to pass all required jobs.
  • Pinned-source MMLU-Pro and GPQA-Diamond converters with source/archive SHA-256 verification, validated caches, deterministic filtering/shuffling, reproducible dataset-v1 ZIPs, and source/license/profile evidence without committing third-party questions.
  • A trusted-local llmbenchlab-evaluate CLI with prepare, run, resume, and report; OpenAI-compatible model discovery and canary preflight; hidden/environment-only API keys; explicit request-bound confirmation; direct database execution; and missing-question recovery. Remote Provider endpoints require HTTPS, while plain HTTP is accepted only for loopback hosts; discovery rejects a model ID that reflects the current Key, and canary rejects a returned model that differs from the requested target.
  • Atomic, non-overwriting terminal Run reports containing a protocol/source/model/execution summary, optional metadata groups, and every persisted per-question Response in paginated JSONL. Report metrics are derived from planned questions plus persisted Responses, and metrics_provenance identifies drift from persisted Run aggregate fields.
  • Web/API write-only Provider credentials: the Models password field accepts an 8–8192-byte visible-ASCII api_key directly, never reads it back, and distinguishes stored, legacy environment, and none sources without displaying the legacy environment-variable name. A one-row-per-model model_credentials table stores only AES-256-GCM ciphertext, nonce, algorithm and key ID; API and Worker share a deployment keyring while the existing environment-variable and trusted-local CLI paths remain compatible.
  • A first-class Evaluation Runs page in the main navigation with all-status history, status filtering, 20-item pagination, manual refresh, two-second polling only while the current page contains pending/running work, and stable links back to Run evidence.
  • Per-Run Web generation controls with Benchmark-aware output/read-timeout recommendations, numeric max_tokens up to 131,072 or null to omit the field and use the Provider default, and read_timeout_seconds from 1 to 1,800 seconds frozen into the execution snapshot. The API/protocol-v1 default remains 256 when max_tokens is omitted; null is not an unlimited-output promise.

Changed

  • Loaded the three already prepared, Git-ignored standard dataset ZIPs into the default personal SQLite through the existing validated import API: GPQA-Diamond (198 questions), MMLU-Pro Direct (12,032), and MMLU-Pro Official-CoT (12,032). This is a local data-maintenance result, not redistributed third-party data or a product/protocol change; the pre-import database backup remains local and no Provider was called. Documentation commit 0163b67c00eb59ae59db5f3adb679ad85c799142 was pushed, and its exact-SHA GitHub Actions run 33266167547 passed all four required jobs.
  • Changed the combined make dev launcher to keep the console quiet after a concise address/log summary. API, Worker, and Vite output is appended to separate Git-ignored artifacts/dev-logs files with UTC session markers and private local permissions; individual service Make targets remain foreground diagnostics, and child failures still propagate their status.
  • Governed production logging at the source boundary: application calls use literal argument-free messages, structured values remain allowlisted and finite, third-party logger text/identity is replaced by fixed classification, and the raw Uvicorn access handler is disabled rather than bypassing the JSON redaction contract.
  • Extended the Phase 2 migration/importer head from 20260827_0004 / 12 tables to 20260828_0005 / 13 tables. A populated worker_processes table blocks 0005 -> 0004 before DDL; only an explicitly emptied table may downgrade, after which the existing governance/audit 0004 guards still apply.
  • Hardened the final P2-06 integrity boundaries: importer reconciliation now compares committed target facts through the same canonical contract, no-op retention mutations still execute post-commit verification, PostgreSQL archive/delete/restore operations hold advisory plus row locks, and audit archives enforce the record-count cap before per-line decoding.
  • OpenAI-compatible Chat Completions now explicitly requests stream:true plus streamed usage, consumes token/heartbeat/usage SSE events through [DONE], and retains a normal-JSON success fallback. read_timeout_seconds is documented as the idle wait for the next response bytes rather than a total generation wall-clock limit.
  • Standardized score, completion_rate, and answered_accuracy as 0–100 values across implementation, API documentation, protocol, and ADR.
  • Configured local CORS for both localhost:5173 and 127.0.0.1:5173, while continuing to reject wildcard origins.
  • Made Run snapshots authoritative for historical model/provider/pricing/execution display, isolated leaderboard ranking by Benchmark protocol/version/Hash, and kept unknown usage or pricing as null instead of silently treating it as zero.
  • Preserved the canonical global→provider→model→run governance lock order during Run admission, and explicitly rolled back rejected/backlog-full transactions before returning typed errors. These fixes removed the PostgreSQL deadlock and retained-lock timeout exposed by the first two exact-SHA capacity attempts.
  • Made Alembic the sole runtime schema owner; backend startup now requires the database at migration head, while setup, migrate, and container startup share the same guarded preflight.
  • Moved Run execution ownership out of the API process. POST /api/v1/runs now commits database truth before a best-effort queue notification; Redis loss does not erase or decide task state.
  • Replaced startup-time failure of all running Runs with lease expiry, database reconciliation, fenced recovery, cancellation convergence, and bounded terminal failure semantics.
  • Kept llmbenchlab-protocol-v1 scoring, completion, accuracy, token, cost, and leaderboard meanings unchanged while adding operational reliability fields to Run snapshots and API responses.
  • Changed Compose from a two-service SQLite demonstration to a PostgreSQL/Redis reliable-development topology. PostgreSQL and Redis remain internal; API and frontend ports bind to loopback by default.
  • Raised the dataset-v1 resource ceiling to 20,000 questions, 128 MiB for questions.jsonl, and 130 MiB for ZIP archives so the pinned 12,032-question MMLU-Pro test split can be imported while retaining line, compression-ratio, path, and schema controls.
  • Reworked Runner question scheduling to at most concurrency consumer tasks, moved large snapshot loading off the event loop so the claimed lease continues heartbeating, reused and explicitly closed one OpenAI-compatible HTTP client per Run, and omitted blank system messages for provider compatibility.
  • Enriched immutable Run benchmark snapshots with schema version, source, license, dimension, and language while keeping llmbenchlab-protocol-v1 scoring and API v1 paths unchanged.
  • Extended the Alembic chain through 20260827_0004 and the stopped SQLite→PostgreSQL importer to all 12 application tables. Encrypted model_credentials move atomically with their Models, while governance policies/scopes/buckets, question execution cursors, Provider-attempt ledger rows and audit events retain their ordering and reconciliation fingerprints. The keyring remains outside the database and must be transferred separately.
  • Managed API Runs now freeze the active policy ID/hash and explicit per-Run input/Token/cost overrides at admission. Adapter retries are governed per actual HTTP attempt, database-clock rate/budget deferrals remain recoverable, confirmed pre-send releases preserve the unsent retry ordinal in a new ledger generation, and trusted-local CLI Runs remain explicitly legacy_unmanaged.
  • Run scheduling now separates failed attempts from cooperative dispatches, yields after a finite question quantum, orders due work for bounded fairness, and exposes typed managed/delayed/exhausted status in API and Run Detail UI. Backlog overflow returns a stable local 429 without losing accepted database truth.
  • Added paginated Run audit and task-history views, database-timestamp queue/execution/end-to-end latency summaries, Provider request/model/fingerprint/finish metadata after strict normalization, and credential lifecycle audit events limited to fixed non-secret fields.
  • Made Run detail evidence truthful for large Benchmarks: the UI reports the API total, pages Responses 100 at a time with global question numbering, and links back to the all-status Run list.

Fixed

  • Fixed managed Runs without an explicit input_token_reservation being permanently exhausted when Provider actual input exceeded a UTF-8 observational estimate. New attempts now leave input reservation and reserved cost unset in that case while preserving Provider actual usage; explicit input/output reservations and reserved cost derived from complete bounds plus frozen prices still enforce overdraw. Added data-only Alembic head 20260830_0007 to recompute only governance_scopes.overdrawn, preserve historical ledger/actual/Response/Run facts, reject upgrade or downgrade while any reservation is active, and restore the old derived predicate on downgrade. Run Detail now uses neutral historical wording for overdraw instead of attributing every case to conservative settlement. The real-Compose acceptance seam now verifies nullable input/cost bounds without coercing a valid null settlement through float().
  • Added ADR-0017 and the forward-only, schema-equivalent Alembic repair revision 20260829_0006 for databases that executed an early 0004 variant before three governance indexes were present. Migration preflight accepts canonical schemas or only a missing subset of those three indexes, so an interrupted SQLite repair is resumable; it backs up SQLite, validates newly historical PostgreSQL 0005 metadata, rejects multiple active policies, and continues to reject every other schema drift.
  • Prevented external LogRecord extras from injecting allowlisted request/run fields, rejected FIFO/non-regular archive inputs without blocking by opening them nonblocking before fstat, and made oversized audit archives fail on their global line cap before decoding attacker-controlled lines.
  • Made importer post-commit verification and no-op retention post-verification fail closed instead of treating an unchanged row count as sufficient evidence; PostgreSQL retention paths now preserve their advisory/row-lock gate through the mutation decision.
  • Made materialized governance scope/minute counters fail closed when they diverge from the never-delete attempt ledger; low or high counter drift cannot be used to bypass admission, mutation, reconciliation, or importer preflight.
  • Prevented a confirmed released_pre_send reservation from consuming a Provider retry ordinal. The old ledger row remains terminal and unique while the next local generation resumes at the first unsent HTTP ordinal, including max_retries=0.
  • Removed a CI-only race from the Runs-page polling tests by waiting for and selecting the page's 2-second interval instead of assuming the most recent global timer belongs to the component; Testing Library may concurrently own a 50-millisecond wait timer.
  • Fixed long generations remaining silent until one final JSON body even though the client used HTTP streaming APIs. Incremental Provider SSE now consumes token/heartbeat bytes as they are actually flushed by the Provider/proxy, while malformed, in-stream-error, oversized, or unterminated streams produce stable per-question errors instead of partial successful answers.
  • Fixed make setup failing with table models already exists after an earlier development startup had created unversioned tables. Supported SQLite layouts are now integrity-checked, consistently backed up, stamped only to their verified revision, and upgraded without dropping existing models, Benchmarks, questions, runs, or responses; unknown/partial schemas are rejected before stamping.
  • Prevented duplicate delivery, stale lease owners, cancellation races, and ACK-result uncertainty from duplicating Responses or changing terminal protocol-v1 aggregates.
  • Distinguished SQLite-import failures before commit (exit 2), an unconfirmed PostgreSQL COMMIT outcome (exit 4), and failures after a confirmed commit (exit 3), so operators are not told to retry data that may already exist.
  • Made the Phase 2 acceptance harness normalize PostgreSQL fractional seconds with 1–6 digits on Python 3.9; the first failing final run and its successful cleanup remain recorded before the corrected 8/8 rerun.
  • Recomputed persisted Response evidence before both fail-attempt and expired-lease dead-letter transitions, preventing a partially completed Failed Run from retaining stale zero aggregates.
  • Made terminal report summaries, groups, and response evidence use one evidence-derived metric source even when legacy/stale Run aggregate fields differ.
  • Let the trusted-local CLI fenced-reclaim an expired, incomplete running lease after reaping terminal evidence, preventing resume from waiting forever for the deliberately stopped regular Worker.
  • Serialized Model credential/endpoint mutation with Run snapshot creation through one dialect-aware lock: PostgreSQL uses SELECT ... FOR UPDATE, while SQLite acquires BEGIN IMMEDIATE before reading the Model.
  • Fixed first-run keyring initialization when a macOS PATH resolves bare python3 to PyPy, whose os.link(..., dir_fd=..., follow_symlinks=False) rejects the bootstrap's secure no-clobber install with EINVAL. Setup/dev/Make entry points now ask uv explicitly for a dependency-free CPython script run, retain the existing symlink/atomicity boundary, retry only transient atomic filesystem errors after confirmed cleanup, and report only a symbolic errno when installation still fails.
  • Classified Provider finish_reason="length" failures as output_truncated when content is missing, empty, or lacks a final parsed answer, instead of reporting only a generic empty/parse/provider-shape failure.
  • Rejected boolean and string coercion for Run output budgets and read timeouts so true cannot silently become one token or one second.
  • Prevented active-page polling from superseding a slower user-triggered page/filter request, and return a shrinking filtered Run list to its last valid page instead of leaving an unreachable empty page.
  • Corrected Web layout clipping and alignment for Benchmark details, immutable snapshots and form field groups; the fifth main-navigation item remains usable across desktop/tablet/mobile, and Runs use readable cards through 1100px without root-page horizontal overflow.

Security

  • Restricted the new exporter to fixed metric families and enum labels with no Run/Model/Provider/Worker/question/reservation IDs, URLs, hashes, origins, correlation IDs, or error text. Audit archives use a strict full-row allowlist and never admit API keys, Authorization/Cookie values, credential ciphertext/nonces/keyring, Provider URLs, prompts, questions, Responses, or raw Provider bodies; archive paths and invalid input are not reflected by CLI errors.
  • Restricted governance and credential audit payloads to typed allowlists and stable opaque identifiers; no API Key, Authorization header, credential ciphertext/nonce, keyring material, Provider URL, question text, prompt, or response body is admitted. Audit/policy/counter corruption fails closed and records only a fixed integrity event in an independent transaction where possible.
  • Bounded streaming Chat responses independently at 64 MiB of SSE wire data, 1 MiB per event, and 4 MiB of aggregated content while preserving the 4 MiB JSON/64 KiB error limits. Raw SSE lines are not logged, and content is aggregated before exact current-Key replacement so a Key split across deltas is still removed.
  • Restricted Benchmark ZIP import by size, entry name/type, compression ratio, schema, and fixed root filenames; dataset contents are never executed.
  • Kept Provider plaintext out of persistence and all read schemas; api_key exists only as a write-only SecretStr request field, while Provider/network errors are bounded and redacted.
  • Rejected Mock remote fields, URL credentials/query/fragment, all unsupported Model default-parameter keys, reflected validation inputs, and non-finite numeric values.
  • Documented that arbitrary compatible-provider URLs remain an SSRF risk and that the unauthenticated MVP must not be exposed publicly.
  • Kept credentialed importer DSNs out of argv via --target-env, rejected passwords in --target, and emitted only row counts and SHA-256 reconciliation digests rather than imported row contents.
  • Limited published Compose ports to loopback and kept PostgreSQL/Redis off the host network by default. This does not add authentication, TLS, tenant isolation, or production hardening.
  • Documented the at-least-once boundary: local database evidence is idempotent, but a Worker crash after a Provider response and before local commit can repeat an upstream call or charge.
  • Kept real API keys out of argv, plaintext persistence, read API responses, reports, and automated tests. Model discovery is identity-only and capped at 2 MiB; Chat JSON success, SSE wire/event/aggregated content, and error bodies are capped at 4 MiB, 64 MiB/1 MiB/4 MiB, and 64 KiB respectively. The exact current Key is removed from successful content, raw usage, Provider request IDs, returned model IDs, system fingerprints, and finish reasons before persistence; model discovery/canary errors remain bounded and sanitized.
  • Required a typed confirmation before any canary or formal request, showed a conservative HTTP-attempt upper bound, rejected active Runs/disabled or conflicting Models before paid preflight, and documented that this is not a Token or monetary budget.
  • Documented the trusted-local exclusivity and SSRF/data-egress boundary: regular API/Worker processes must be stopped before direct CLI execution, and arbitrary compatible-provider URLs remain unsuitable for untrusted/public use.
  • Made Web/API keys write-only and short-lived in browser state: the password field is cleared when submission starts, on close/provider switch, and on unmount; pending writes are aborted on close/unmount, exact reflected error text is redacted, and no Key is written to browser storage or console. Credential status is exposed through credential_source/has_api_key while the legacy environment-variable-name field remains compatible; reads never expose plaintext or encryption material.
  • Bound each AES-256-GCM envelope to the Model ID and normalized Provider origin; changing origin requires a new Key, and any Provider endpoint/credential change is rejected while that Model has a pending or running Run. Missing/invalid keyrings, unknown key IDs and authentication failures fail closed with stable non-secret errors.
  • Kept the new credential store within the trusted loopback boundary: it adds neither authentication nor a production KMS, and compromise of both the database and deployment keyring can recover Provider keys.
  • Prevented a new or preserved stored Key from being copied from the credential flow into any field of the exact ModelRead projection or the Run snapshot's derived model sub-projection during create/PATCH. Preserved values are decrypted only for this fail-closed comparison; Provider JSON evidence is recursively redacted, including numeric Key values in usage/token and status fields, before persistence.
  • Allowed an unreadable stored envelope, including an unknown or retired key_id, to be repaired by an isolated explicit new Key or removed by an isolated switch to Mock/legacy environment mode. Recovery requests that also change unrelated public fields fail with a stable 422; requests that preserve stored without a valid replacement fail with a stable 503; both leave the transaction unchanged.
  • Ignored client-controlled X-Request-ID values and generated a fresh server UUID for every API response, preventing a caller from duplicating a write-only Key into a reflected/logged correlation header.

Verification

  • P2-06 implementation gates are complete. Commit 9a20676dcf545040782f04c166205d0043345753 was pushed to codex/complete-evaluation-workflow and is tracked by PR #3; exact-SHA GitHub Actions run 33164609388 passed all four required jobs. Its local gates include green combined targeted suites, make lint across 152 Python files plus ESLint/TypeScript, backend 916 passed, 33 skipped, frontend 38 passed, offline Mock smoke 1 passed, 7 deselected, PostgreSQL 16/Redis 7 migration/check and 33 passed, 0 skipped integration, isolated SQLite migration/check, frontend build, Compose config, eight-rule Prometheus v3.5.0 validation, and a 76-file technical/security review with 0 Blocker/High/Medium. Clean-SHA Compose acceptance passed 9/9 with .pytest_cache/artifacts/phase2-acceptance/llmbenchlab-p2-92e173eeee28/evidence.json (SHA-256 e4ffb8668fd3fa62d59b5d83f5c29eede35b327d88e6099345acd5950670fc47), Worker gauges 2/2/2/0/0, and empty container/volume/network cleanup. Clean capacity passed with .pytest_cache/artifacts/phase2-capacity/llmbenchlab-p2-ca5673061b0f/evidence.json (SHA-256 2382f9138f09028f269d76c341b236dd4089d678c8a2323582045fac2b4f5039): 1W/2W/burst QPS 7.267474/12.962228/9.333604, wall 8.255963/4.628834/6.428385s, 18 Runs/270 Responses/270 question executions/271 reservations/1230 audit events, zero question error/drift/duplicate/PEL/lag, Worker expected 2 with shortfall 0, and empty container/volume/network/image cleanup with image counters 1/1/0/0. The earlier dirty acceptance .pytest_cache/artifacts/phase2-acceptance/llmbenchlab-p2-11554c25ec2d/evidence.json (SHA-256 d5f058457dbc29875cbac4bc38345b810b5ed556ea538862d309116ceb629fde) and dirty capacity .pytest_cache/artifacts/phase2-capacity/llmbenchlab-p2-c6de062ab77e/evidence.json (SHA-256 4aeb8271dd81e8671fc287942839f8d06862140ea9a6bf1d7ee5660265aa8453, 1229 audit events) remain historical evidence rather than being overwritten. These are offline Mock observations, not production or real-Provider SLOs. Evidence-documentation commit ec2959680459a14aa308bd4d9ebcc6bb7bfcf3a6 was pushed, and its exact-SHA GitHub Actions run 33165775037 passed all four required jobs, completing P2-06. Phase 2 stays in_progress; P2-07 now has a planned work package but remains unimplemented. The default user SQLite was below head, so its direct alembic check failed and it was deliberately not migrated.
  • Recorded two corrected local gate invocations without hiding their first outcomes: the initial integration cleanup command was rejected by the local safety policy before any container started, then the explicit cleanup/run passed; an over-broad Ruff invocation over scripts/ surfaced 93 existing modernization warnings, while the intended --select E,F,I correctness/import gate passed.
  • On the governance/audit candidate at 665244e095905083b606b8e98e946ed1a02dc0fc, make test passed with 604 passed, 29 skipped on the backend and 38 passed on the frontend; a separate real PostgreSQL/Redis integration run passed 29/29. That frozen implementation also passed make lint, offline make smoke (1 passed, 7 deselected), the frontend production build, isolated SQLite and PostgreSQL migration/check gates, docker compose config --quiet, targeted governance/API/Worker suites, enhanced capacity, and 9/9 Compose acceptance.
  • Pushed the implementation and lock-order fixes through exact SHA 665244e095905083b606b8e98e946ed1a02dc0fc. Its enhanced capacity evidence (40deadeb…0588) records finite policy, exact 4×202 + 2×429, cooperative/cross-Model fairness, fault recovery, 18 Runs/270 Responses/271 reservations/1229 audit events, and empty cleanup; its 9/9 acceptance evidence (ab311665…ddec) includes all three deterministic database seams and empty cleanup. GitHub Actions run 33099260233 passed all four required jobs. At that gate, the formal SLO/exporter/retention/backup/Worker-progress closure was still pending.
  • On the later P2-01 v2 implementation SHA b6a35fef1dd069ebb54b69955058915c722aa34d, make test passed with 829 passed, 29 skipped on the backend and 38 passed on the frontend. GitHub Actions run 33146681285 passed all four required jobs; a fresh 1+5 formal invocation then passed 23/23 SLO checks and every child hard invariant/cleanup check, with aggregate SHA-256 a76d167bb664e2ee3ee7514c39ac738b76cef37776d7b66e1175a8596329d0d9. Evidence-documentation commit 875f13a253c40b7573d45c6287385e60f2bb8f04 was pushed and its exact-SHA GitHub Actions run 33150080341 passed all four required jobs, completing P2-01 repository closeout for this fixed Mock-only single-host profile. Phase 2 remains in_progress; P2-06 is completed, and P2-07 is planned but not implemented.
  • Opened PR #1 for the trusted-local evaluation/Web credential/Run UX/SSE branch and obtained all four required GitHub Actions jobs for exact SHA ab15862eab4870dda01fb079b44b509a7d737627 in run 33078921254; this closes the previously documented no-PR/no-CI boundary for that SHA only.
  • Passed 205 backend non-infrastructure tests, five real PostgreSQL/Redis integration tests, 13 frontend tests, the isolated offline Mock smoke test, Ruff/ESLint/TypeScript checks, Vite production build, and SQLite/PostgreSQL Alembic upgrade/check/downgrade/upgrade gates.
  • Passed an isolated default-build Compose acceptance harness in all eight scenarios: topology/readiness, protocol-v1 baseline, API restart during execution, exact lease-owner SIGKILL and natural takeover, Redis stop/start with database reconciliation, pending cancellation, running cancellation plus duplicate delivery, and PostgreSQL head -> 0001 -> head protocol round-trip.
  • Verified final Redis consumer-group pending=0 and lag=0, unchanged three-point canonical hashes for one 15-question baseline Run's protocol-v1 core fields and its 15 Responses, and cleanup with no project containers, volumes, or networks left behind. This hash is not a whole-database snapshot.
  • Verified the SQLite importer against PostgreSQL 16 for success, pre-commit rollback, two-source contention, commit acknowledgement loss, post-commit snapshot/output failure, secret-safe CLI behavior, and cleanup of every random target database and temporary container.
  • Passed the final Web-credential current-worktree local gate: 427 backend tests passed with 6 explicit infrastructure skips when DSNs were absent; all 6 real PostgreSQL/Redis integration tests passed separately; 21 frontend tests, lint/typecheck, production build, offline Smoke, PostgreSQL Alembic upgrade/check, lock/config checks, and the isolated 8/8 Compose fault acceptance passed. The keyring bootstrap has 24 focused security regressions, and a fresh first-write probe also passed with PyPy placed first in PATH because the entry point selected CPython through uv.
  • Downloaded and validated the complete pinned sources locally: both MMLU-Pro profiles produced 12,032 questions and GPQA-Diamond produced 198; small llmbenchlab-evaluate prepare --limit 2 runs also passed through the public CLI. No real Provider or API key was used.
  • The Web-credential base implementation b19bdac9236f9b2f927166ebe30578ced3d9f53e and bootstrap remediation d26cdbe4f3f97057ce09d5d7a539ddbfe605d967 were pushed normally. The remediation SHA has no GitHub Actions run because the branch has no pull request and the workflow listens only to pull requests/main; local success is not being reported as a substitute.
  • Web credential automation uses only marker keys, fixed test keyrings, MockTransport/stub fetch and Mock evaluation. No real Provider was called. Local gates are complete; stage commit/push and exact-SHA CI remain separate remote gates and are not inferred from local success.
  • Trusted-loopback browser verification confirmed a password input, no api_key_env control, no post-save Key echo, and no fake browser Key in application logs.
  • The Web Run-UX/generation-budget slice passed its final local full-worktree gates (442 passed, 6 skipped backend; 36 passed frontend) with Mock/stub regression coverage and 390–1280px breakpoint checks. Feature commit 467d0243b4fb081c2d637b20ee0958c3bd6ee6d1 was pushed normally; its exact-SHA Actions query and branch PR query both returned empty because the workflow listens only to pull requests/main and this branch has no PR. No real Provider was called, and local success is not reported as remote CI success.
  • The OpenAI-compatible SSE slice passed its final local full-worktree gates: 50 focused Adapter tests; 453 passed, 6 skipped backend and 36 passed frontend through make test; lint/format/typecheck, production build, offline Smoke, Alembic, lock, Compose config, diff and high-confidence secret checks. Feature commit af345af1048eeddffd784fdca1da419df95da7e2 was pushed normally; its exact-SHA Actions query and branch PR query both returned empty because the workflow listens only to pull requests/main and this branch has no PR. All Provider behavior used MockTransport/stubs; the post-fix llama.cpp/Cloudflare/Caddy path was not called, so local success is not reported as proof that the user's real proxy chain is fixed or as remote CI success.