Summary
There is currently no way to bound what an LLM-backed scan will spend before starting it. skillspector scan --llm either completes or does not, and the cost is only knowable afterwards — if at all. I would like a cap that is enforced during the scan, plus a way to see the projected cost before committing to it.
Motivation
Unlike an API-key provider with a prepaid balance, the CLI providers (claude_cli, codex_cli, gemini_cli) draw on the user's personal subscription. There is no per-scan meter, no spending alert, and no natural stopping point — the cost surfaces later as consumed quota.
That asymmetry matters for how SkillSpector gets adopted. A scanner that might cost an unknown amount is one users run rarely and reluctantly; a scanner that can be told "spend at most N calls on this" is one they can wire into a pre-install gate or a timer.
Concretely: I disabled the LLM stage entirely in my automation, not because it lacked value — it caught a natural-language exfiltration skill that the static stage scored 0 findings on — but because I could not bound what an unattended run would cost. The static-only stage I fell back to is precisely the one blind to that attack class. A cap would have let me keep the layer that works.
Scan cost also scales with the unit, not with the user's intent: pointing --changed at a marketplace containing hundreds of skills is one command, and there is no signal beforehand that it is three orders of magnitude more expensive than the last one.
Proposed change
Two independent pieces; the first is the useful half if only one lands.
1. A hard cap, enforced mid-scan
--max-llm-calls N stop issuing LLM calls after N
--llm-token-budget N stop after N total tokens (needs #242)
Plus env equivalents (SKILLSPECTOR_MAX_LLM_CALLS, SKILLSPECTOR_LLM_TOKEN_BUDGET) so timers and CI can set them without rewriting the command.
2. A projection, before spending
--estimate report planned LLM call count and exit without calling
Even a crude count — analyzer slots × eligible components — is enough to distinguish "4 calls" from "400" and is what makes the cap easy to choose.
Behaviour when the cap is hit
This is the part I would ask for care on, because there is a trap adjacent to it that this project already knows about.
A scan stopped by its budget must not be reportable as complete. #290 covers the analogous failure — a deep scan reporting SAFE when the provider was unavailable — and a budget cap creates a second, more frequent path to the same wrong conclusion: an unfinished scan that looks clean because the LLM stage never reached the risky component.
Suggested: mark the report explicitly, e.g. metadata.llm_budget_exhausted: true alongside the existing llm_available / llm_calls_attempted, and have the risk assessment refuse to assert a clean verdict on an unfinished scan. A non-zero exit code for "stopped early" would let CI distinguish it from "finished, nothing found".
Whether the cap should be per-scan or per-skill in recursive mode is a real design question I do not have a strong view on. Per-skill avoids one large skill starving the rest; per-scan is what a user's wallet actually cares about.
Related
Summary
There is currently no way to bound what an LLM-backed scan will spend before starting it.
skillspector scan --llmeither completes or does not, and the cost is only knowable afterwards — if at all. I would like a cap that is enforced during the scan, plus a way to see the projected cost before committing to it.Motivation
Unlike an API-key provider with a prepaid balance, the CLI providers (
claude_cli,codex_cli,gemini_cli) draw on the user's personal subscription. There is no per-scan meter, no spending alert, and no natural stopping point — the cost surfaces later as consumed quota.That asymmetry matters for how SkillSpector gets adopted. A scanner that might cost an unknown amount is one users run rarely and reluctantly; a scanner that can be told "spend at most N calls on this" is one they can wire into a pre-install gate or a timer.
Concretely: I disabled the LLM stage entirely in my automation, not because it lacked value — it caught a natural-language exfiltration skill that the static stage scored 0 findings on — but because I could not bound what an unattended run would cost. The static-only stage I fell back to is precisely the one blind to that attack class. A cap would have let me keep the layer that works.
Scan cost also scales with the unit, not with the user's intent: pointing
--changedat a marketplace containing hundreds of skills is one command, and there is no signal beforehand that it is three orders of magnitude more expensive than the last one.Proposed change
Two independent pieces; the first is the useful half if only one lands.
1. A hard cap, enforced mid-scan
Plus env equivalents (
SKILLSPECTOR_MAX_LLM_CALLS,SKILLSPECTOR_LLM_TOKEN_BUDGET) so timers and CI can set them without rewriting the command.2. A projection, before spending
Even a crude count — analyzer slots × eligible components — is enough to distinguish "4 calls" from "400" and is what makes the cap easy to choose.
Behaviour when the cap is hit
This is the part I would ask for care on, because there is a trap adjacent to it that this project already knows about.
A scan stopped by its budget must not be reportable as complete. #290 covers the analogous failure — a deep scan reporting SAFE when the provider was unavailable — and a budget cap creates a second, more frequent path to the same wrong conclusion: an unfinished scan that looks clean because the LLM stage never reached the risky component.
Suggested: mark the report explicitly, e.g.
metadata.llm_budget_exhausted: truealongside the existingllm_available/llm_calls_attempted, and have the risk assessment refuse to assert a clean verdict on an unfinished scan. A non-zero exit code for "stopped early" would let CI distinguish it from "finished, nothing found".Whether the cap should be per-scan or per-skill in recursive mode is a real design question I do not have a strong view on. Per-skill avoids one large skill starving the rest; per-scan is what a user's wallet actually cares about.
Related
llm_calls_attempted: 4in the report was accurate while saying nothing about what had actually been spent.