fix(run6.3): smoke-found bench harness fixes (7 schema/protocol bugs) - #120
Merged
Conversation
…cape .env localhost interference Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When invoked as `uv run python scripts/bench.py`, sys.path[0] is `scripts/` (the script dir), so `from scripts.bench_ingest import ingest_fixture` fails because the parent (repo root) is not on path. Add it explicitly. Smoke-found: bench failed at phase 5 with `ModuleNotFoundError: No module named 'scripts'` after the workspace+key bootstrap succeeded.
…duction schema match) Replace created_at with started_at in all extraction_runs and integrator_runs queries; pipeline_traces.created_at is unchanged. Update test synthetic schema to drop created_at from extraction_runs/integrator_runs and add missing columns that the production SELECT references. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…at always NULL); workspace isolation suffices extraction_runs.started_at is nullable with no server_default and no code path sets it — WHERE started_at >= :ts filters out every row. Bench creates a fresh workspace per run so workspace_id scoping is sufficient. Temporal filter retained only on pipeline_traces.created_at and integrator_runs.started_at (both always populated). Repurposed test to cover pipeline_traces temporal filter instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, predicate_definitions) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Run 6.3 manual smoke fixes. Running
just bench --fixture smallend-to-end against a live stack (withANTHROPIC_API_KEYset) caught 7 real bugs inscripts/bench.pyandscripts/bench_report.pythat were invisible to the existing unit-test suite (which uses a synthetic SQLite schema with whatever names the queries expect — i.e. doesn't catch production-schema mismatches).Result: smoke now passes end-to-end. exit 0, all 9 phases complete,
bench_results/{date}-{sha}/contains validsummary.md+manifest.json(env_summary masked) + extraction_runs/integrator_runs/pipeline_traces JSON dumps.Bugs fixed (in commit order)
c07c9e9phase_bootstrapdidn't sendslugonPOST /workspaces(required byWorkspaceCreateschema). Also:.env'sALAYA_DATABASE_URL=...localhost...interferes withdocker compose upbecause compose's${VAR:-default}interpolation reads.env, overriding the compose-internal default.127.0.0.1:5432.5783f99from scripts.bench_ingest import ...inside functions failed because whenpython scripts/bench.pyruns, sys.path[0] isscripts/, not the repo root.ModuleNotFoundError: No module named 'scripts'.2dd7e9fbench_report.pySELECTedcreated_atfromextraction_runsandintegrator_runs, but those tables havestarted_at(nocreated_atcolumn). Also SELECTed nonexistentcortex_cost_usd/crystallizer_cost_usdandentities_merged(the latter is in extraction_runs but bench_report queriedintegrator_runs).column "entities_merged" does not exist.6ad3ff1extraction_runs.started_atis nullable and never populated by any code path — temporal filterstarted_at >= :tsexcluded all rows. Workspace isolation per-bench-run already provides scoping.result: "empty_workspace".3a5b656bench_report.pyqueriedclaims,entities,predicates, but the actual table names arel2_claims,l1_entities,predicate_definitions.relation "claims" does not exist.6aed181bench_report.pyJOIN'dintegrator_actionsonintegrator_run_id, but the actual column isrun_id.column ia.integrator_run_id does not exist.abaa7bf_serialize_rowhandled UUID and datetime, but PostgresNUMERICcolumns returnDecimal, whichjson.dumpsrejects.Object of type Decimal is not JSON serializable.c07c9e9phase_integratoronly accepted HTTP 200/201 fromPOST /integrator-runs/trigger, but the router returns 202 Accepted.FAIL: integrator trigger returned 202.Why unit tests didn't catch any of this
tests/test_bench_report.pybuilds an in-memory SQLite database with synthetic CREATE TABLE statements that match whatever the queries expect — it doesn't validate against production schema. The only way to catch these was an actual end-to-end smoke run against Postgres + the real worker pipeline. Lesson for Run 6.4: add an integration-marked test that runsformat_summaryagainst the real postgres testcontainers schema (the same fixture used bytests/test_migration_009_forward_down.py).Smoke evidence (after this PR)
Manifest excerpt:
{ "result": "complete", "exit_code": 0, "extraction_runs_count": 5, "integrator_runs_count": 1, "pipeline_traces_count": 13, "env_summary": { "ANTHROPIC_API_KEY": "set", ... all others "unset" }, "model_versions": { all "unset" }, "cache_hit_ratio_per_stage": { "cortex": 0.0, "crystallizer": 0.0, "integrator:*": null }, "quality_proxies": { "claims_per_event_stddev": 0.0, "dedup_actions": 0, "run_failure_count": 0, ... } }grep -lE "ak_[a-zA-Z0-9_-]{16,}" bench_results/.../*returns empty — no API key leaked into any artifact.Verification (gates)
Test plan
just bench --fixture mediumshows non-zerocost_usd(medium fixture uses real Anthropic calls; cortex/crystallizer should generate cost)just bench --fixture medium --cache-warm-check— second-run cortexcache_hit_ratio > 0.5(SC5)Known limitations / follow-ups (not in scope of this PR)
format_summaryagainst real testcontainers Postgres (with all tables migrated). Tracking idea:RUN6.3.FU.03— bench_report integration test against live schema.extraction_runs.started_atnever populated: this is a pre-existing schema/code-path inconsistency unrelated to bench. The bench-side fix is to scope by workspace_id only. A future cleanup could decide whether to populatestarted_atinExtractionRunRepository.update_status(when transitioning to processing) or drop the column.recalc_usageran but adapter cost flow needs verification on a larger fixture. Not blocking — small is for smoke; medium/large will give real numbers.🤖 Generated with Claude Code