feat(eval): report quality-constrained localization cost - #427
Conversation
github-actions
Bot
added
type/docs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c833068dc4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "claude-haiku-4-5-20251001" | ||
| ], | ||
| "channel": "anthropic-api-direct", | ||
| "token_semantics": "prompt_excludes_cache", |
There was a problem hiding this comment.
When this bundled snapshot is used to reprice CodeNib cells produced through LiteLLM, Anthropic cache hits are overbilled: LiteLLM documents prompt_tokens as including cache-hit input tokens (https://docs.litellm.ai/docs/completion/prompt_caching), but prompt_excludes_cache makes project_cell_cost() leave those tokens in uncached_input and also add them again as cache_read_input whenever cache_read_input_tokens > 0. This inflates projected USD for cached runs and can change the constrained optimum, so the Anthropic/LiteLLM snapshot needs semantics that subtract cached reads (or a separate read/write-aware mode).
Useful? React with 👍 / 👎.
| required = { | ||
| "model", | ||
| "instance_id", | ||
| "query_id", |
There was a problem hiding this comment.
When users pass a complete output directory from the existing run_sweep() path, every cell is skipped as a non-cell because those records include instance_id, subset_id, model, and rep but no query_id (checked codenib/eval/agent_runner/sweep.py), while the new docs say complete sweep directories are valid inputs. That makes the CLI fail with no agent-runner cells found for those sweep artifacts; either emit a stable query id for run_sweep() cells (for example the instance id) or narrow the loader/docs to query-sweep outputs only.
Useful? React with 👍 / 👎.
| if not _looks_like_cell(value): | ||
| skipped_non_cells += 1 |
There was a problem hiding this comment.
When a query sweep records an instance-level load failure, that failure lives only in query_sweep_summary.json (checked the run_query_sweep() load-exception path), but the new loader discards every non-cell JSON file here. As a result, a complete sweep directory can silently drop all planned attempts for a failed repository, so strict denominators still pass and the report contradicts its infrastructure_failures_score_zero_quality contract by biasing cost and quality toward the instances that loaded successfully.
Useful? React with 👍 / 👎.
Persist prompt, completion, cache, and cost provenance fields in agent-runner cells, including schema-complete failure records. Correct the provider-specific cache token documentation and retain the LiteLLM calculator version for runtime estimates.\n\nVerified with focused runner tests, the full unit tier, and pre-commit.
Add strict recursive cell and retry audits, repository-clustered paired quality guards, token and USD cost per successful localization, and constrained arm selection. Bundle an immutable direct-provider pricing schema while keeping runtime estimates and local-model prices explicit.\n\nValidated against the retained five-model 500-query study without model calls. The report records 8,500 input cells, 7,500 analyzed cells, and explicit exclusion of unrelated Haiku arms.\n\nVerified with 2552 unit tests, pre-commit, strict docs, a built wheel, and an installed console-script smoke.
c833068 to
437e955
Compare
Summary
Adds an auditable cost-per-success evaluation path for repository localization. The report applies a paired localization-quality guard before choosing the least-cost policy, preserves raw usage provenance, and treats unpriced or incomplete usage as unavailable rather than free.
Closes #426.
Changes
Type of Change
Testing
Validated with:
pytest -m "not slow and not integration and not integration_serial and not integration_serial_consumer" -x --tb=short(2552 passed, 10 skipped, 183 deselected)pre-commit run --all-filesmkdocs build --strictandpython scripts/check_public_docs.pycodenib-quality-cost-reportsmoke on the retained 500-query Gemini result setChecklist