Follow-up to PR #346 (thread #346 (comment)).
The accuracy-only setup in commands/benchmark/execute.py currently forces num_workers=1, max_connections=1, target_concurrency=1 for every accuracy-only run of every workload to satisfy the edge compliance single_stream gate. This (a) silently throttles existing accuracy workflows (DeepSeek-R1, GPQA, AIME) to single-stream with no opt-out, and (b) inverts the checker's purpose by rewriting the persisted config.yaml so the gate can't fail.
Proposed fix (per @viraatc):
- Add a
single_stream: bool field to HTTPClientConfig (endpoint_client/config.py) with a --single-stream cyclopts alias.
- Resolve it in the model validator to
num_workers=1, max_connections=1; make it mutually exclusive with explicitly setting num_workers/max_connections (validator raises if both given).
- Edge YAML sets
settings.client.single_stream: true (already has target_concurrency: 1).
- Delete the accuracy-only override block, the mirrored log-only branch in
_run_benchmark_async, and TestSetupBenchmarkAccuracySingleStream.
Net effect: persisted config.yaml honestly reflects what ran, the compliance gate validates rather than being satisfied by a rewrite, and non-edge accuracy runs keep their throughput.
Follow-up to PR #346 (thread #346 (comment)).
The accuracy-only setup in
commands/benchmark/execute.pycurrently forcesnum_workers=1, max_connections=1, target_concurrency=1for every accuracy-only run of every workload to satisfy the edge compliancesingle_streamgate. This (a) silently throttles existing accuracy workflows (DeepSeek-R1, GPQA, AIME) to single-stream with no opt-out, and (b) inverts the checker's purpose by rewriting the persistedconfig.yamlso the gate can't fail.Proposed fix (per @viraatc):
single_stream: boolfield toHTTPClientConfig(endpoint_client/config.py) with a--single-streamcyclopts alias.num_workers=1, max_connections=1; make it mutually exclusive with explicitly settingnum_workers/max_connections(validator raises if both given).settings.client.single_stream: true(already hastarget_concurrency: 1)._run_benchmark_async, andTestSetupBenchmarkAccuracySingleStream.Net effect: persisted
config.yamlhonestly reflects what ran, the compliance gate validates rather than being satisfied by a rewrite, and non-edge accuracy runs keep their throughput.