fix: cap max_tokens on scan requests, and read the API key from the env - #31
Merged
Merged
Conversation
Two problems found while running `modelfuzz scan` against hosted models
through OpenRouter for the first time.
`scan` never set max_tokens, so a gateway reserves the target model's full
context window up front. Against anthropic/claude-sonnet-5 every request
failed:
402 - This request requires more credits, or fewer max_tokens.
You requested up to 65536 tokens, but can only afford 3937.
The same scan with max_tokens=1024 completes and returns a real verdict. A
probe only needs room for one tool call and a mutation for one prompt, so
1024 is generous; --max-tokens raises it. This made scanning a hosted model
fail outright on exactly the credit-limited accounts most first-time users
have, which is the worst possible audience for it.
Separately, --api-key was a plain option with no envvar, so a real key could
only reach it as a command-line argument -- landing in shell history and
visible in `ps` for the duration of the run. It now reads MODELFUZZ_API_KEY.
Verified end to end against OpenRouter with no key on the command line:
openai/gpt-4o-mini 3/3 seeds broke through at generation 1
anthropic/claude-sonnet-5 0/3, refused through generation 3
Tests 71 -> 76. The stub client now records max_tokens per request, so both
call sites are asserted rather than just the probe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #30 β merge that first and this diff shrinks to just the two changes below. Version β
0.3.5.Both problems were found by actually running
modelfuzz scanagainst hosted models through OpenRouter for the first time. Neither is theoretical.1.
scanwas unusable against hosted models_probeand_mutatenever setmax_tokens, so a gateway reserves the target model's full context window up front. Againstanthropic/claude-sonnet-5, every single request failed:Proven to be the cause, not a credit problem β same model, same prompt, same key:
This made hosted scanning fail outright on credit-limited accounts β which is exactly what a first-time user has. Now capped at
1024by default, overridable with--max-tokens. A probe needs room for one tool call and a mutation for one prompt, so 1024 is generous.2. The API key could only reach the CLI on the command line
--api-keywas a plaintyper.Optionwith noenvvar, so a real key had to be typed as an argument β landing in shell history and visible inpsfor the duration of the run. It now readsMODELFUZZ_API_KEY, and Typer documents it in--helpautomatically:Verified end to end
Real scans against OpenRouter, no key on any command line:
openai/gpt-4o-minianthropic/claude-sonnet-5Tests
71 β 76.
StubClientnow recordsmax_tokensper request, so both call sites are asserted rather than just the probe. New coverage: env var is read, an explicit--api-keystill wins over it, the dummy-key fallback still works for local models, and--max-tokensis honoured.One trap worth noting for future tests: the new cases originally used the default non-empty mutation, which keeps every lineage alive until the wall-clock budget expires β four tests Γ 30s hung the suite. They use
mutation=""to end each lineage after one mutate call, which still exercises both call sites. Suite is back to 0.03s.Found but NOT fixed here
Running this surfaced a separate, more serious problem: against an aligned model the mutation loop degenerates. The mutation step asks the target to rewrite the attack; Claude refuses, and its refusal text becomes the next generation's "attack prompt". So generations 2 and 3 probe with refusal prose, not attacks. The run reported
7 attack attemptswhen only 3 were genuine.That needs a design decision (detect refusals and kill the lineage, or use a separate model as the mutation engine), so it is recorded in the local audit backlog rather than bolted on here.