Skip to content

Move bfcl-eval to an out-of-process uv subproject (remove parent-env pip caveats) #390

Description

@Palanivelg

Context

Raised in PR #346 review (pyproject.toml:78).

bfcl-eval is currently a parent-env optional dependency (the [bfcl] extra). uv-side isolation ([tool.uv].conflicts + the filelock/virtualenv CVE floors) is airtight for uv sync, but it leaks for pip, which ignores both uv.lock and conflicts:

  • we had to relax numpy==2.4.4 -> numpy>=1.26.4 (pyproject.toml:78), so pip install -e . no longer pins numpy reproducibly;
  • pip install -e ".[bfcl,dev]" hard-fails on the filelock floor.

This is a regression from the repo's exact-pin convention, forced by needing bfcl-eval installable in the parent env.

Proposal

Move bfcl-eval out of the parent env into an out-of-process uv run --project subproject, mirroring the existing pattern for:

  • VBench (examples/09_Wan22_VideoGen_Example/accuracy/)
  • DeepSeek-R1 (src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/)

bfcl-eval is a strong fit: it's already lazy-imported everywhere (bfcl_v4_scorer.py:42, multi_turn.py:144), and single-turn loading is bfcl-eval-free on the R2 parquet path (pre-formatted messages+tools) - only scoring (ast_checker) pulls it in. Multi-turn needs bfcl-eval in-process for live stateful execution (bfcl_v4_execution.py:37), but it's already its own CLI (bfcl_v4_multi_turn_cli), a natural subproject entrypoint.

Payoff

Moving scoring to a subproject would let the parent:

  • restore numpy==2.4.4 (exact pin);
  • drop the [bfcl] extra, the [tool.uv].conflicts block, and the filelock/virtualenv CVE floors;
  • remove the pip reproducibility caveats without changing the accuracy gate.

Scope

Deferred out of PR #346 to keep it focused on landing the workload + gate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions