Context
Raised in PR #346 review (pyproject.toml:78).
bfcl-eval is currently a parent-env optional dependency (the [bfcl] extra). uv-side isolation ([tool.uv].conflicts + the filelock/virtualenv CVE floors) is airtight for uv sync, but it leaks for pip, which ignores both uv.lock and conflicts:
- we had to relax
numpy==2.4.4 -> numpy>=1.26.4 (pyproject.toml:78), so pip install -e . no longer pins numpy reproducibly;
pip install -e ".[bfcl,dev]" hard-fails on the filelock floor.
This is a regression from the repo's exact-pin convention, forced by needing bfcl-eval installable in the parent env.
Proposal
Move bfcl-eval out of the parent env into an out-of-process uv run --project subproject, mirroring the existing pattern for:
- VBench (
examples/09_Wan22_VideoGen_Example/accuracy/)
- DeepSeek-R1 (
src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/)
bfcl-eval is a strong fit: it's already lazy-imported everywhere (bfcl_v4_scorer.py:42, multi_turn.py:144), and single-turn loading is bfcl-eval-free on the R2 parquet path (pre-formatted messages+tools) - only scoring (ast_checker) pulls it in. Multi-turn needs bfcl-eval in-process for live stateful execution (bfcl_v4_execution.py:37), but it's already its own CLI (bfcl_v4_multi_turn_cli), a natural subproject entrypoint.
Payoff
Moving scoring to a subproject would let the parent:
- restore
numpy==2.4.4 (exact pin);
- drop the
[bfcl] extra, the [tool.uv].conflicts block, and the filelock/virtualenv CVE floors;
- remove the
pip reproducibility caveats without changing the accuracy gate.
Scope
Deferred out of PR #346 to keep it focused on landing the workload + gate.
Context
Raised in PR #346 review (pyproject.toml:78).
bfcl-evalis currently a parent-env optional dependency (the[bfcl]extra). uv-side isolation ([tool.uv].conflicts+ the filelock/virtualenv CVE floors) is airtight foruv sync, but it leaks forpip, which ignores bothuv.lockandconflicts:numpy==2.4.4->numpy>=1.26.4(pyproject.toml:78), sopip install -e .no longer pins numpy reproducibly;pip install -e ".[bfcl,dev]"hard-fails on thefilelockfloor.This is a regression from the repo's exact-pin convention, forced by needing
bfcl-evalinstallable in the parent env.Proposal
Move
bfcl-evalout of the parent env into an out-of-processuv run --projectsubproject, mirroring the existing pattern for:examples/09_Wan22_VideoGen_Example/accuracy/)src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/)bfcl-evalis a strong fit: it's already lazy-imported everywhere (bfcl_v4_scorer.py:42,multi_turn.py:144), and single-turn loading isbfcl-eval-free on the R2 parquet path (pre-formatted messages+tools) - only scoring (ast_checker) pulls it in. Multi-turn needsbfcl-evalin-process for live stateful execution (bfcl_v4_execution.py:37), but it's already its own CLI (bfcl_v4_multi_turn_cli), a natural subproject entrypoint.Payoff
Moving scoring to a subproject would let the parent:
numpy==2.4.4(exact pin);[bfcl]extra, the[tool.uv].conflictsblock, and the filelock/virtualenv CVE floors;pipreproducibility caveats without changing the accuracy gate.Scope
Deferred out of PR #346 to keep it focused on landing the workload + gate.