Skip to content

fix(run6.3): smoke-found bench harness fixes (7 schema/protocol bugs) - #120

Merged
egerev merged 7 commits into
mainfrom
fix/run6.3-bench-smoke
Apr 25, 2026
Merged

fix(run6.3): smoke-found bench harness fixes (7 schema/protocol bugs)#120
egerev merged 7 commits into
mainfrom
fix/run6.3-bench-smoke

Conversation

@egerev

@egerev egerev commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Run 6.3 manual smoke fixes. Running just bench --fixture small end-to-end against a live stack (with ANTHROPIC_API_KEY set) caught 7 real bugs in scripts/bench.py and scripts/bench_report.py that were invisible to the existing unit-test suite (which uses a synthetic SQLite schema with whatever names the queries expect — i.e. doesn't catch production-schema mismatches).

Result: smoke now passes end-to-end. exit 0, all 9 phases complete, bench_results/{date}-{sha}/ contains valid summary.md + manifest.json (env_summary masked) + extraction_runs/integrator_runs/pipeline_traces JSON dumps.

Bugs fixed (in commit order)

# Commit Bug Symptom
1 c07c9e9 phase_bootstrap didn't send slug on POST /workspaces (required by WorkspaceCreate schema). Also: .env's ALAYA_DATABASE_URL=...localhost... interferes with docker compose up because compose's ${VAR:-default} interpolation reads .env, overriding the compose-internal default. Phase 4: 400 validation error; Phase 2: migrations container fails to connect to 127.0.0.1:5432.
2 5783f99 from scripts.bench_ingest import ... inside functions failed because when python scripts/bench.py runs, sys.path[0] is scripts/, not the repo root. Phase 5: ModuleNotFoundError: No module named 'scripts'.
3 2dd7e9f bench_report.py SELECTed created_at from extraction_runs and integrator_runs, but those tables have started_at (no created_at column). Also SELECTed nonexistent cortex_cost_usd/crystallizer_cost_usd and entities_merged (the latter is in extraction_runs but bench_report queried integrator_runs). Phase 8: column "entities_merged" does not exist.
4 6ad3ff1 extraction_runs.started_at is nullable and never populated by any code path — temporal filter started_at >= :ts excluded all rows. Workspace isolation per-bench-run already provides scoping. Phase 8 succeeded but result: "empty_workspace".
5 3a5b656 bench_report.py queried claims, entities, predicates, but the actual table names are l2_claims, l1_entities, predicate_definitions. Phase 8: relation "claims" does not exist.
6 6aed181 bench_report.py JOIN'd integrator_actions on integrator_run_id, but the actual column is run_id. Phase 8: column ia.integrator_run_id does not exist.
7 abaa7bf _serialize_row handled UUID and datetime, but Postgres NUMERIC columns return Decimal, which json.dumps rejects. Phase 8: Object of type Decimal is not JSON serializable.
extra (in commit 1) c07c9e9 phase_integrator only accepted HTTP 200/201 from POST /integrator-runs/trigger, but the router returns 202 Accepted. Phase 7: FAIL: integrator trigger returned 202.

Why unit tests didn't catch any of this

tests/test_bench_report.py builds an in-memory SQLite database with synthetic CREATE TABLE statements that match whatever the queries expect — it doesn't validate against production schema. The only way to catch these was an actual end-to-end smoke run against Postgres + the real worker pipeline. Lesson for Run 6.4: add an integration-marked test that runs format_summary against the real postgres testcontainers schema (the same fixture used by tests/test_migration_009_forward_down.py).

Smoke evidence (after this PR)

$ just bench --fixture small  (with ANTHROPIC_API_KEY set)
[bench] Phase 1: preflight             — OK
[bench] Phase 2: docker compose up     — OK (compose env override)
[bench] Phase 3: API health            — healthy
[bench] Phase 4: bootstrap workspace   — Workspace created (slug provided)
[bench] Phase 5: ingest small.jsonl    — 5 events queued
[bench] Phase 6: drain                 — 5/5 terminal
[bench] Phase 7: integrator            — completed (202 accepted)
[bench] Phase 8: report                — manifest + summary + JSON dumps written
[bench] Phase 9: teardown              — services stopped

Manifest excerpt:

{
  "result": "complete",
  "exit_code": 0,
  "extraction_runs_count": 5,
  "integrator_runs_count": 1,
  "pipeline_traces_count": 13,
  "env_summary": { "ANTHROPIC_API_KEY": "set", ... all others "unset" },
  "model_versions": { all "unset" },
  "cache_hit_ratio_per_stage": { "cortex": 0.0, "crystallizer": 0.0, "integrator:*": null },
  "quality_proxies": { "claims_per_event_stddev": 0.0, "dedup_actions": 0, "run_failure_count": 0, ... }
}

grep -lE "ak_[a-zA-Z0-9_-]{16,}" bench_results/.../* returns empty — no API key leaked into any artifact.

Verification (gates)

uv run ruff check .              — clean
uv run ruff format --check .     — clean
uv run pyright                   — 0 errors
uv run pytest                    — 1158 passed, 1 skipped, 84 deselected
uv run python scripts/bench.py --help — works
just bench --fixture small       — exit 0, complete artifact (smoke evidence above)

Test plan

  • CI green
  • After merge: just bench --fixture medium shows non-zero cost_usd (medium fixture uses real Anthropic calls; cortex/crystallizer should generate cost)
  • Optional: just bench --fixture medium --cache-warm-check — second-run cortex cache_hit_ratio > 0.5 (SC5)

Known limitations / follow-ups (not in scope of this PR)

  • Test coverage gap: unit tests still use synthetic SQLite schema. A future PR should add an integration-marked test that runs format_summary against real testcontainers Postgres (with all tables migrated). Tracking idea: RUN6.3.FU.03 — bench_report integration test against live schema.
  • extraction_runs.started_at never populated: this is a pre-existing schema/code-path inconsistency unrelated to bench. The bench-side fix is to scope by workspace_id only. A future cleanup could decide whether to populate started_at in ExtractionRunRepository.update_status (when transitioning to processing) or drop the column.
  • Run-level cost shows $0.000000 on small fixture: likely because recalc_usage ran but adapter cost flow needs verification on a larger fixture. Not blocking — small is for smoke; medium/large will give real numbers.

🤖 Generated with Claude Code

egerev and others added 7 commits April 26, 2026 00:58
…cape .env localhost interference

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When invoked as `uv run python scripts/bench.py`, sys.path[0] is `scripts/`
(the script dir), so `from scripts.bench_ingest import ingest_fixture`
fails because the parent (repo root) is not on path. Add it explicitly.

Smoke-found: bench failed at phase 5 with `ModuleNotFoundError: No module
named 'scripts'` after the workspace+key bootstrap succeeded.
…duction schema match)

Replace created_at with started_at in all extraction_runs and integrator_runs
queries; pipeline_traces.created_at is unchanged. Update test synthetic schema
to drop created_at from extraction_runs/integrator_runs and add missing columns
that the production SELECT references.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…at always NULL); workspace isolation suffices

extraction_runs.started_at is nullable with no server_default and no code path
sets it — WHERE started_at >= :ts filters out every row. Bench creates a fresh
workspace per run so workspace_id scoping is sufficient. Temporal filter retained
only on pipeline_traces.created_at and integrator_runs.started_at (both always
populated). Repurposed test to cover pipeline_traces temporal filter instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, predicate_definitions)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot added the tests label Apr 25, 2026
@egerev
egerev merged commit 8335434 into main Apr 25, 2026
12 checks passed
@egerev
egerev deleted the fix/run6.3-bench-smoke branch April 25, 2026 17:20
egerev added a commit that referenced this pull request Apr 25, 2026
Move Run 6.3 row from active Core Stabilization section to "Runs already
completed". Add Run 6.1, 6.2, and 6.3 to the completed list with PR numbers
and merge dates. Follow-up PRs #115#120 plus 2026-04-26 follow-ups noted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant