Fix is_file_ready validation and isolate bug-validator probes from fm_agent - #141
Fix is_file_ready validation and isolate bug-validator probes from fm_agent#141Fancyyyf wants to merge 5 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87185758cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| When the project under validation is FM-Agent itself, the probe **must not start or call FM-Agent to validate FM-Agent**. In particular, do not invoke `run_pipeline()`, `run_incremental_pipeline()`, `main.py`, the FM-Agent CLI, OpenCode, or any subprocess/entry point that starts an FM-Agent workflow. Test only the smallest relevant unit with mocks or fixtures. If the reported behavior cannot be tested without starting an FM-Agent workflow, classify it as `NOT CONFIRMED`; do not run the workflow from the probe. | ||
|
|
||
| Never use the active repository, its isolation snapshot, the current working directory, or its `fm_agent/` directory as a probe workspace. Any probe-created files must live under a fresh temporary directory owned by the probe. The probe must not delete, replace, or regenerate the active run's `fm_agent/` artifacts. |
There was a problem hiding this comment.
In the FM-Agent self-validation path, this mandatory guard now says any probe-created files must live under a fresh temp directory and not under fm_agent/, but the same prompt requires fm_agent/bug_validation/probe_<bug_id>.<ext> and _validate_single_bug only treats fm_agent/bug_validation/<bug_id>.result.json as success. A validator that follows this guard can place the probe/result outside fm_agent/bug_validation, causing the pipeline to retry and then report a missing result artifact even though validation ran; the guard needs an explicit exception for the required output artifacts while keeping any probe workspace/test fixtures isolated.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e66349b87
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| #### FM-Agent self-validation guard (mandatory) | ||
|
|
||
| When the project under validation is FM-Agent itself, the probe **must not start or call FM-Agent to validate FM-Agent**. In particular, do not invoke `run_pipeline()`, `run_incremental_pipeline()`, `main.py`, the FM-Agent CLI, OpenCode, or any subprocess/entry point that starts an FM-Agent workflow. Test only the smallest relevant unit with mocks or fixtures. If the reported behavior cannot be tested without starting an FM-Agent workflow, classify it as `NOT CONFIRMED`; do not run the workflow from the probe. |
There was a problem hiding this comment.
When the mismatch is in an FM-Agent internal helper, this guard now forbids invoking main.py, the CLI, or a workflow, but the still-mandatory entry-point rule below requires loading only through the public entry point. Since this repository's package initializer is empty and many src/* helpers are not exported by the console entry point, a validator cannot both obey the new guard and exercise those units, so real self-validation bugs can be forced to NOT CONFIRMED; add an explicit exception allowing direct target-module imports for FM-Agent unit probes.
Useful? React with 👍 / 👎.
1 participant
This change improves FM-Agent self-validation safety and correctness:
Tightens is_file_ready() to require complete, correctly ordered [SPEC] and [INFO] comment blocks.
Rejects marker text found inside ordinary source code, incomplete headers, mixed comment prefixes, and source code inserted between header blocks.
Updates the bug-validator prompt to forbid using fm_agent/ as a probe or test directory.
Requires generated probes to use isolated temporary directories and prevents probes from recursively triggering FM-Agent self-validation.