Score combined RAG development cohorts with one checkpoint load - #554
Score combined RAG development cohorts with one checkpoint load#554gonzalobenegas wants to merge 2 commits into
Conversation
|
Claude encountered an error after 0s —— View job I'll analyze this and get back to you. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 221f799432
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| mendelian="results/scores/{model}/mendelian_traits.parquet", | ||
| complex="results/scores/{model}/complex_traits.parquet", | ||
| sge="results/scores/{model}/sge.parquet", |
There was a problem hiding this comment.
When an existing model registration is switched to rag_combined, these outputs publish biologically different, retrieval-context scores under the same shared-S3 keys previously owned by compute_scores, replacing or reusing artifacts whose paths still imply the legacy semantics. The pipeline guidance explicitly requires task-specific output namespaces and preservation of existing S3-backed paths, so route this backend through a distinct namespace rather than changing the producer behind the existing keys. .agents/skills/develop-snakemake-pipelines/SKILL.mdL33-L36
Useful? React with 👍 / 👎.
|
|
||
| def rag_harness_input(wildcards): | ||
| uri = get_model_config(wildcards.model)["rag_harness"]["uri"] | ||
| return storage.s3(uri) if uri.startswith("s3://") else uri |
There was a problem hiding this comment.
For a valid local rag_harness.uri, this returns a bare path, but the checked-in workflow profile enables default-storage-provider: s3; normal invocations therefore resolve the path beneath the default S3 prefix instead of reading the local file and report a missing remote input. The new workflow test masks this by explicitly using --default-storage-provider none; return local(uri) for this branch, as the repository's other mixed local/S3 input helper does.
Useful? React with 👍 / 👎.
1 participant
RAG development records have variable retrieval coverage, so their human windows occur at different token positions and cannot use the reference-window transform.
This adds an explicitly registered combined backend that loads one checkpoint for Mendelian, Complex Traits, and SGE, then restores each benchmark's canonical row order and existing score output path.
The backend validates pinned cohort identities, alleles, coordinates, and group metadata before inference.
It uses fixed padded batches, applies RC within species segments, gathers human-only LLR/JSD positions, and pools the final human-token states for the existing grouped probes.
The reference-only backend remains the default.
Validation: the locked evals_v2 CI suite passed 418 tests with 5 skips, including real-model parity with the existing prefix-sharing kernel, padding invariance, joint-versus-separate prediction and canonical routing, and both combined and reference-only subset Snakemake dry-runs.
Repository pre-commit checks pass, including Ruff 0.16.2 and Snakefmt 2.0.3.
Independent review findings were addressed and the follow-up review found no remaining issues.
GPU bf16/compilation parity and full-context throughput remain experiment pilot checks before biological inference.
No model is registered or evaluated by this PR.
Closes #553.