Repository navigation
Enforce successful funding checks and record evaluation limitations - #2
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7fd618afef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const citations = [...answer.matchAll(/\[source:([^\]\r\n]+)\]/g)].map( | ||
| (match) => match[1], | ||
| ); |
There was a problem hiding this comment.
The regex ignores malformed markers rather than exposing them to the gate as the comment intends. For example, an answer containing one valid citation plus [source:] or an unterminated [source:FAKE records only the valid ID, so citationsPass remains true; a no-match answer containing only such a marker also passes. Detect any [source: marker that does not match the exact citation syntax and fail citation validation so malformed model output cannot produce a successful evaluation.
Useful? React with 👍 / 👎.
The approved 15-case evaluation scored 97.9% weighted but missed a required eligibility call. The previous weighted-only gate still passed it. This change enforces successful retrieval and eligibility checks in both generation and streaming, keeps missing applicant/partner facts unclear and makes tool/retrieval/citation failures fail the evaluation regardless of average score.
Validation: 48 offline tests pass, including eight tests through the actual SDK loops with model/retrieval mocks and fetch prohibited. Clean install, typecheck, lint and production build pass; full npm audit reports zero advisories. The immutable paid-run artifact records clean commit8fd08a1, corpus/vector/case hashes, models, timing and usage. It predates the control fixes: no second paid evaluation or post-fix behavioral pass is claimed.
Credentialed evaluation is manual and fails when keys are absent. No deployment or provider secrets are included.