[2.0] Add new Frontier-CS 2.0 problem vllm_llm_serving_optimization (#145)

* feat: Add new Frontier-CS 2.0 problem vllm_llm_serving_optimization

* feat: Add real SWE-bench resolved fraction

* feat(2.0/vllm): BFCL-memory workload + single-H100 contention + scoring/gate fixes

Updates the vLLM serving-latency task with this iteration's work:
- BFCL switched from AST function-calling to the multi-turn `memory` category
  (vendored kv backend + 5 pre-baked per-scenario snapshots); real, non-zero,
  non-ceilinged accuracy used as a guardrail.
- Continuum job-FCFS reference.patch added (wins ~1.1-1.6x on SWE, single H100).
- Single H100 (was H100:2): creates the KV contention scheduling needs; H100:2
  was measured to over-provision (no queueing -> codex ~1.0x).
- BFCL at jps=1.0 + down-weighted to 0.2 (SWE 0.8): measured to have no
  reproducible latency signal at any arrival rate (batch-numerics non-determinism
  swamps it), so it mainly serves as correctness gate + accuracy guardrail.
- BFCL correctness gate uses a 5% abs-OR-rel tolerance to absorb that
  non-determinism instead of falsely failing good patches.
- modal_app serves Qwen3-Coder-30B-A3B; dynamic serving_harness label; misc fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: momoway

2.0/README.md

@@ -47,6 +47,21 @@ applies the submitted patch to a clean skeleton, runs a hidden arena against
 multiple baseline bot families, and scores by mean baseline win rate with a
 small faster-win tiebreak. The online generals.io service is not used.
 
+## vLLM LLM-Serving Optimization
+
+This systems problem asks agents to patch a clean upstream vLLM checkout to
+reduce the end-to-end latency of an LLM serving system on a multi-turn agentic
+workload, while keeping accuracy near a baseline. Its problem ID is
+`vllm_llm_serving_optimization`. The served model is
+`meta-llama/Llama-3.1-8B-Instruct` on a single Modal L40S, and the workload is a
+mini-swe-agent SWE-bench run. The agent submits a Python-only patch and can run
+an async public test (a subset of the final eval set) that returns real latency
+and accuracy feedback. Scoring is the geometric-mean latency speedup versus a
+vanilla-vLLM baseline, gated by an accuracy guardrail: accuracy within 5% of the
+baseline does not affect the score, and beyond that the score decays
+inverse-proportionally with the accuracy drop. Like duckdb-e2e, the agent and
+judge run in separate Docker environments.
+
 ## BBOPlace ISPD2005
 
 This VLSI placement problem asks agents to generate macro placement candidates

2.0/problems/vllm_llm_serving_optimization/.dockerignore

@@ -0,0 +1,8 @@
+**/__pycache__
+**/*.pyc
+harbor/app/.public_test
+docs
+docker/README.md
+*.md
+reference.patch
+harbor/app/solution.patch

2.0/problems/vllm_llm_serving_optimization/DESIGN.md

@@ -0,0 +1,124 @@
+# vllm_llm_serving_optimization — Design & Changes
+
+Agent task: patch clean upstream **vLLM v0.11.0** (Python-only, allowlisted
+files) to cut end-to-end latency of an H100 Modal serve of
+`meta-llama/Llama-3.1-8B-Instruct`, preserving generation quality.
+
+## What this revision adds
+
+### 1. BFCL as a second judged workload (50/50 with SWE-bench)
+
+`serving_eval/bfcl.py` runs the **Berkeley Function Calling Leaderboard**
+`simple` (Python) category as a serving workload: one chat completion per
+instance (prompt mode, no native tool-calling), client-side per-instance
+latency, and a deterministic **AST-equality** correctness check against a
+ground-truth call. The data slice is vendored under `serving_eval/bfcl_data/`
+(`BFCL_v4_simple_python.json` + `possible_answer/…`, from `bfcl-eval==2026.3.23`,
+Apache-2.0) so the judge runs **real, offline** evaluation with no network and no
+heavy `bfcl_eval` dependency.
+
+Why BFCL: an 8B model resolves ~0% of SWE-bench Verified, so the old accuracy
+guardrail was mathematically dead (`baseline_accuracy = 0 ⇒ multiplier ≡ 1`).
+Llama-3.1-8B gets a **meaningfully non-zero** BFCL `simple` accuracy, so its
+guardrail is *live*.
+
+The self-contained decoder + checker were **cross-validated** against
+`bfcl_eval`'s `ast_checker` on all 400 records: 395/400 agree, and on the 5
+nested dict/list cases ours is only ever *more* lenient (never stricter) — so it
+is symmetric and fair for baseline-vs-patched (and identical for both sides).
+Wrong-function and prose outputs are correctly scored incorrect.
+
+Scoring (`scoring.py`, `evaluator.full_evaluation`):
+```
+final_score = swebench_weight * swe_score + bfcl_weight * bfcl_score   # 0.5 / 0.5
+workload_score = clip(100*log2(geomean(per_instance_speedup)), 0, 100) * accuracy_multiplier
+```
+
+### 2. Reference solution (`reference.patch`) — continuum job-level FCFS + long-prefill cap
+
+Two-file, self-contained, correctness-preserving change (both files are in the
+strongly-allowed `vllm/v1/core/sched/**`):
+
+- **`request_queue.py` — `JobFCFSRequestQueue` (the continuum soul).** The WAITING
+  queue is ordered by each conversation's **job_id first-arrival time** instead of
+  per-request arrival time, so a later turn of an in-flight conversation is
+  admitted ahead of a brand-new job's first prefill — keeping ongoing multi-turn
+  work moving and reusing its (already cache-hot) prefix. It is activated by
+  default via `create_request_queue` (FCFS policy → `JobFCFSRequestQueue`); no
+  launch-flag change is needed. Requests without a job_id fall back to plain
+  per-request FCFS, so it is a safe drop-in.
+- **`scheduler.py` — long-prefill admission cap.** Caps fresh *uncached* long
+  prefills per step (deferred via the existing skip-and-requeue when decode work
+  is in flight), so a burst of new long prompts cannot head-of-line-block decode.
+
+**job_id plumbing (no protocol/request.py changes).** The workload runners
+(`agent_runner.py`, `bfcl.py`) send a stable per-conversation id via
+`extra_body={"vllm_xargs": {"job_id": <instance_id>}}`. v0.11.0 already forwards
+`vllm_xargs` into `sampling_params.extra_args`, which vanilla vLLM ignores and the
+reference reads as `request.sampling_params.extra_args["job_id"]`. So the same
+requests serve identically on the baseline; only the patched scheduler uses the
+signal. Ordering uses `request.arrival_time` only (never wall-clock), so it is
+deterministic and changes only admission *order* — never tokens — and the greedy
++ BFCL correctness gates pass.
+
+This is materially more faithful to continuum than a client-signal-free version:
+continuum's headline is exactly job-level FCFS keyed on job_id (KV-pinning and the
+tool-call-length estimator are the parts it stubs/omits).
+
+**Evidence it beats baseline.** This is the same mechanism a real codex trial
+agent used to measure **1.79× geomean speedup** on the 30-instance SWE-bench
+slice against this exact baseline (the reference diffs from blob `2b2cd63`, which
+matches the trial patch's base). The win comes from smoothing prefill bursts so
+each scheduler iteration keeps the running decode batch flowing (lower p50/p95
+inter-token latency) and from letting hot-prefix conversations resume without
+queueing behind a cold long prefill. On the blended metric the SWE-bench half
+improves strongly; the BFCL half (short single-turn prompts, no long prefills to
+defer) is roughly neutral, so the reference still scores clearly above the
+0-point baseline.
+
+To re-validate live (needs Modal + HF creds and a free H100):
+```
+MODAL_TOKEN_ID=… MODAL_TOKEN_SECRET=… FRONTIER_SUBMISSION_ROLE=final \
+  python3 evaluator.py reference.patch         # judge path (baseline vs patched)
+# or, agent-side: bash harbor/app/public_test.sh run
+```
+
+### 3. Audit fixes folded in
+
+- **Live accuracy guardrail** via BFCL (above) — the headline correctness fix.
+- **Real, non-zero, always-runs correctness eval**: BFCL AST scoring needs no
+  Docker or swebench harness, so it never silently degrades to a proxy and is
+  never all-zero.
+- **BFCL per-sample correctness gate** (`measure._bfcl_correctness_ok`): a
+  temperature-0 patch may not flip BFCL answers correct→wrong/undecodable
+  (tolerates `bfcl_max_correctness_regressions` flips for batch-numerics noise).
+- **Anti-inflation scoring** (`scoring.paired_speedups`): per-instance speedup
+  clamped to `[1/cap, cap]` (cap = 8); a patched instance that errored/early-exited
+  is counted as a regression (`1/cap`), so "fail fast" can no longer inflate the
+  geomean.
+- **Binary-hunk patch-policy bypass closed** (`evaluator.validate_patch`): patches
+  containing `GIT binary patch` / `Binary files … differ` are rejected (the +line
+  token scanner can't see a base85 payload; the build is Python-only anyway).
+- **Build-timeout mismatch fixed**: `evaluation.build_timeout_seconds` is now 7200,
+  matching the documented budget and `environment.build_timeout_seconds`.
+
+## Layout / build
+
+The task source was reconstructed (`serving_eval/*.py` recovered from the judge
+image; `evaluator.py`/`config.yaml`/`readme`/`harbor/app` from the generated
+dataset). `docker/build_images.sh` does an **overlay rebuild** of the agent +
+judge images (`experimental-v0.11.1`), replacing `/opt/serving_eval` with the
+refreshed harness (incl. `bfcl_data/`) and asserting the BFCL data is present in
+the image. Both images carry `/opt/serving_eval`: the judge runs the
+authoritative measurement, the agent image runs the same harness for the public
+test.
+
+## Known limitations
+
+- The SWE-bench sandbox is `LocalSandbox` (empty dir) unless Docker-in-Docker is
+  available on the judge; its accuracy stays a proxy. BFCL now carries the real
+  task-quality guardrail, which is the point of the 50/50 split.
+- Latency is still a single sample per build on independently-autoscaled Modal
+  serves; the cap + per-workload geomean + live guardrail reduce, but do not
+  eliminate, run-to-run variance. Pinning `max_containers=1` and repeated
+  sampling remain future hardening.

2.0/problems/vllm_llm_serving_optimization/config.yaml

@@ -0,0 +1,139 @@
+tag: systems
+runtime:
+  language: python
+  timeout_seconds: 21600
+  environment: "Patched vLLM (v0.11.0) source; Modal H100 GPU serving Qwen3-Coder-30B-A3B-Instruct; two agentic workloads (mini-swe-agent SWE-bench + BFCL memory) scored 50/50; latency-primary judge with a live BFCL accuracy guardrail + greedy/per-sample correctness gates"
+  apt_packages:
+    - bash
+    - ca-certificates
+    - curl
+    - git
+    - python3
+    - python3-pip
+  judge_apt_packages:
+    - bash
+    - ca-certificates
+    - curl
+    - git
+    - python3
+    - python3-pip
+  judge_pip_packages:
+    - modal
+    - openai
+    - datasets
+    - huggingface-hub
+    - rank-bm25
+  docker:
+    # Experimental local images. Build them with
+    # 2.0/problems/vllm_llm_serving_optimization/docker/build_images.sh before running a
+    # local Harbor trial. Both images need a clean upstream vLLM v0.11.0 checkout
+    # (NOT the continuum fork). The judge image additionally vendors the
+    # mini-swe-agent harness and the latency/accuracy scorer.
+    image: frontiercs/vllm-serving-optimization-agent:experimental-v0.11.1
+    judge_image: frontiercs/vllm-serving-optimization-judge:experimental-v0.11.1
+environment:
+  cpus: 8
+  memory_mb: 32768
+  storage_mb: 65536
+  build_timeout_seconds: 7200
+evaluation:
+  # Model + accelerator served on Modal. Qwen3-Coder-30B-A3B (latest Qwen3 coder,
+  # MoE: 30B total / ~3.3B active per token) actually resolves SWE-bench (the 8B
+  # Llama was ~0) and is fast thanks to the sparse MoE. SINGLE H100 on purpose:
+  # the 30B weights (~61GB bf16) leave only ~12-20GB KV (~5 concurrent 32K seqs),
+  # which is exactly the contention this scheduling task needs. H100:2 was measured
+  # to OVER-PROVISION (no queueing -> scheduling can't help -> codex ~1.0x); on one
+  # card the continuum reference wins (SWE ~1.1-1.6x). vLLM TP size is derived from
+  # "H100:N". Avoid B200 (sm_100) — the v0.11.0 precompiled wheel has no Blackwell kernels.
+  model: Qwen/Qwen3-Coder-30B-A3B-Instruct
+  gpu: "H100"
+  # Workload 1 (latency-primary): mini-swe-agent on SWE-bench Lite (split test).
+  # Lite (300, self-contained) is cheaper/cleaner than Verified; instances are
+  # drawn by a fixed-seed RANDOM sample across all 12 repos (not a sorted prefix,
+  # which clustered into 1-2 hard repos and was unrepresentative).
+  dataset: princeton-nlp/SWE-bench_Lite
+  dataset_split: test
+  # Fixed seed for the deterministic random instance sample (SWE + BFCL), so the
+  # evaluated set is reproducible run-to-run.
+  sample_seed: 20260624
+  # Iterative (agent-role) public test: a strict subset of the final eval set.
+  public_slice: "0:5"
+  # Final (verifier-role) evaluation: 50 instances sampled from Lite with
+  # sample_seed (set "0:300" for the full Lite split; that is many H100-hours).
+  eval_slice: "0:50"
+  # Workload 2 (agentic + real correctness): BFCL **memory** category. Each
+  # instance is a multi-turn agentic task — the model is given a key-value memory
+  # tool suite pre-seeded (vendored snapshots) with facts from a prior
+  # conversation, asked a question, and must issue retrieve/search tool calls
+  # over several turns, then answer (word-boundary match vs ground truth). Gives a
+  # genuine multi-step request path AND a non-zero, non-ceilinged accuracy signal
+  # (a strong model gets ~70-90%, not a pinned 1.0), so the guardrail has range.
+  bfcl_public_slice: "0:8"
+  # Full memory verify set: all 155 vendored instances at final.
+  bfcl_eval_slice: "0:155"
+  bfcl_max_tokens: 768
+  memory_max_steps: 20
+  # BFCL memory arrives as a seeded Poisson process at its OWN rate (bfcl_jps).
+  # Measured: BFCL's short, 93%-prefix-cacheable requests have NO clean scheduling
+  # signal at any rate. Below KV saturation (jps<=1.0) the metric is reproducible
+  # (~1.0x, identical run-to-run) but flat; at/above saturation (jps>=1.5) a real
+  # ~1.4x job-FCFS effect appears but is swamped by batch-numerics non-determinism
+  # (the SAME patch measured 0.71x and 1.55x across two clean jps=1.5 runs; an
+  # identical-build control swung 10x at jps=2.5). So BFCL runs at jps=1.0 — the
+  # one reproducible point — mainly for its correctness gate + accuracy guardrail,
+  # and is DOWN-WEIGHTED in the latency blend (see *_weight below). SWE carries the
+  # latency signal (its many-turn latency is robust to single-token flips).
+  bfcl_jps: 1.0
+  # bfcl_workers only caps the fallback burst path (arrival_mode != jps).
+  bfcl_workers: 64
+  # Final score = swebench_weight * SWE-bench score + bfcl_weight * BFCL score.
+  # SWE-heavy: SWE carries the reliable latency signal; BFCL is down-weighted to
+  # 0.2 because at its one reproducible load (jps=1.0) it is latency-neutral (~1.0x)
+  # and mainly serves as the correctness gate + accuracy guardrail.
+  swebench_weight: 0.8
+  bfcl_weight: 0.2
+  # Poisson arrival workload (jobs/second). Mirrors a realistic serving load.
+  arrival_mode: jps
+  jps: 0.5
+  workers: 8
+  step_limit: 50
+  temperature: 0.0
+  max_completion_tokens: 2048
+  # Latency aggregation + scoring.
+  latency_metric: mean_e2e_seconds
+  # Per-instance speedup is clamped to [1/cap, cap]; a failed/early-exit patched
+  # instance is counted as a regression, so "fail fast" cannot inflate the geomean.
+  max_per_instance_speedup: 8.0
+  # Accuracy guardrail (per workload). Within `accuracy_tolerance` relative drop
+  # of baseline => no penalty; beyond it the score decays inverse-proportionally.
+  # No penalty if the accuracy drop is within EITHER the relative tolerance OR
+  # the absolute tolerance (resolve_rate over a finite slice is coarse/noisy).
+  accuracy_tolerance: 0.05
+  accuracy_abs_tolerance: 0.05
+  agent_accuracy_mode: patch_validity
+  final_accuracy_mode: resolve_rate
+  # Greedy-output correctness smoke (fixed prompts must match the baseline
+  # token-for-token at temperature 0 before timing is considered).
+  correctness_smoke_prompts: 8
+  # BFCL per-sample correctness gate: a temperature-0 patch should not flip BFCL
+  # answers correct->wrong/undecodable. Allowed flips = max(the count floor below,
+  # abs_tolerance * n_instances, rel_tolerance * n_baseline_correct) — a 5%/5%
+  # abs-OR-rel band that absorbs batch-numerics non-determinism (which flips many
+  # instances run-to-run even between identical builds under concurrency).
+  bfcl_max_correctness_regressions: 1
+  bfcl_correctness_abs_tolerance: 0.05
+  bfcl_correctness_tolerance: 0.05
+  # Modal serving knobs.
+  modal_scaledown_seconds: 900
+  modal_startup_timeout_seconds: 1200
+  server_health_timeout_seconds: 1800
+  # Per-phase wall-clock budgets (seconds). Matches the documented build budget.
+  build_timeout_seconds: 7200
+  instance_timeout_seconds: 1200
+  # Use a baseline (vanilla vLLM) cached in the judge image when available,
+  # otherwise the judge serves vanilla once and caches it for the trial.
+  baseline_cache_path: /opt/vllm-baseline/baseline_metrics.json
+submission:
+  kind: file
+  path: /app/solution.patch
+  max_queue_size: 2

2.0/problems/vllm_llm_serving_optimization/docker/README.md

@@ -0,0 +1,74 @@
+# Experimental vLLM Serving-Optimization Images
+
+This task needs two images, mirroring the duckdb-e2e split: a public **agent**
+image and a private **judge** image. Both bundle a clean upstream vLLM checkout
+and the shared `serving_eval` harness. Build them before running a local Harbor
+trial:
+
+```bash
+bash 2.0/problems/vllm_llm_serving_optimization/docker/build_images.sh
+```
+
+Defaults:
+
+```text
+VLLM_REF=v0.11.0
+AGENT_TAG=frontiercs/vllm-serving-optimization-agent:experimental-v0.11.0
+JUDGE_TAG=frontiercs/vllm-serving-optimization-judge:experimental-v0.11.0
+```
+
+The agent image contains:
+
+```text
+/app/vllm                 # clean upstream vLLM (no continuum, no reference fix)
+/opt/serving_eval         # shared harness, used by the async public test
+/opt/vllm-baseline        # optional precomputed baseline cache
+```
+
+The judge image contains:
+
+```text
+/opt/vllm-clean           # clean upstream vLLM (build + baseline reference)
+/opt/serving_eval         # shared harness, used by the evaluator
+/opt/vllm-baseline        # baseline-metrics cache (filled on first measurement)
+```
+
+## Runtime requirements (important)
+
+Unlike duckdb-e2e, this task does **not** run the model inside the container.
+Both the agent public test and the judge serve `meta-llama/Llama-3.1-8B-Instruct`
+on a **Modal L40S** built from the (patched) vLLM source. The containers
+therefore need:
+
+- `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` in the environment (Modal auth). Use a
+  Modal service-user token for unattended runs.
+- A Modal Secret named `huggingface-secret` containing `HF_TOKEN` with access to
+  the gated Llama-3.1 weights (`modal secret create huggingface-secret HF_TOKEN=...`).
+  The container also reads `HF_TOKEN` for the `datasets` download.
+- The judge additionally needs a reachable Docker daemon (mounted socket or
+  DinD) to run the SWE-bench per-instance testbeds for the final resolve-rate.
+  When no daemon is reachable, the harness falls back to a local sandbox and the
+  patch-validity accuracy proxy.
+
+The Modal image build uses `VLLM_USE_PRECOMPILED=1`, so only vLLM's Python layer
+is rebuilt from the submitted source (minutes, not a full CUDA compile). This is
+why the patch policy is Python-only.
+
+## Baseline cache
+
+The judge measures the vanilla (clean-tree) baseline once per role and caches it
+at `/opt/vllm-baseline/baseline_metrics.json`, keyed by role (`agent` / `final`).
+Baseline and patched builds are never served simultaneously, so a single L40S is
+sufficient per environment. To precompute and bake the baseline into the image
+(recommended for faster trials), run the harness against the clean tree offline
+and copy the resulting `baseline_metrics.json` into the image at that path.
+
+## Smoke test
+
+```bash
+bash 2.0/problems/vllm_llm_serving_optimization/docker/smoke_images.sh
+```
+
+This checks that the clean vLLM checkout, the `serving_eval` package, and the
+Modal/OpenAI/datasets (and, for the judge, swebench + docker) clients are
+importable. It does not exercise Modal or a GPU.