feat(scan): backend fallback chain (issues #7, #35)

[GeorgelPreput]

What & why

The README advertises a 6-tier backend preference order, but the code never walked it: PickBackend returned exactly one backend and Scan failed closed to SUSPICIOUS the moment it errored. This implements the backend fallback chain discussed in #7 and reported in #35.

• Backend fallback: try next backend in chain before failing closed #35 (HaleTom): a failed Claude CLI immediately failed closed even though Codex was installed — it was never tried. Now the chain tries the next backend first.
• Resume feature for dealing with rate limits #7 (multi-backend half): a rate-limited Claude subscription now falls through to a local model / another backend instead of dying on the first request.

How it works

The chain is environment-derived backends first (a pinned AURSCAN_BACKEND, otherwise all auto-detected backends in the documented order), then ~/.config/aurscan/llm1.conf … llmN.conf in numeric order, de-duplicated. On any per-backend failure — transport/exec error, timeout, or non-genuine output — aurscan prints a one-line WARNING: backend <x> failed; trying next and moves on. The first genuine verdict (real JSON yielding a known OK/SUSPICIOUS/MALICIOUS) wins. Only when the whole chain is exhausted does it fall closed to SUSPICIOUS — unchanged from today.

Strictly additive. With no llmN.conf and a healthy backend, behaviour is byte-for-byte identical: only the first backend is called on success, total failure still fails closed with the same verdict/exit code, and a lone backend failing prints no "trying next" noise. The chain only ever adds attempts on failure.

llmN.conf schema (flat key = value, no new dependency)

key	meaning
backend	claude · codex · api · openai · or a /path/to/exe (custom command, mirroring AURSCAN_BACKEND)
model	model id for api / codex / openai
url	endpoint override (openai /chat/completions, or an Anthropic-compatible /v1/messages gateway for api)
fallback	secondary openai URL (intra-backend)
api_key	bearer / x-api-key for this backend

Values are literal; #/; start a comment only at the start of a line. Secrets are better kept in env vars — aurscan warns on startup if an api_key-bearing file is group/other-readable.

Notable design point

Chain fall-through is decided by a structural "genuine verdict" signal (the parser reports whether real JSON produced a known verdict string), not the model-controlled "(fail-closed)" summary substring the previous code sniffed. Otherwise a backend returning {"verdict":"unsure"} (downgraded to SUSPICIOUS) would wrongly stop the chain.

Testing

• make test (go vet ./... + go test ./...) — green. New: internal/config/config_test.go (parser edge cases: numeric ordering, BOM/CRLF, =-in-value, literal values, unreadable-file skip, perm warning), internal/scan/chain_test.go (transport/parse/unknown-verdict fall-through, genuine-verdict-stops-chain, cmd fall-through, claude→codex auto-detect, spec-URL precedence, secret-no-leak, exhaustion, single-backend silence), and a pipeline no-backend regression. All pass under -race.
• Live matrix on the compiled binary: openai→openai (fake key → real key) and openai→claude fall-through each produced a real verdict; a single fake-key backend failed closed to SUSPICIOUS; secret redaction confirmed (hasKey:yes, no key in --debug output).

Deferred to follow-ups (flagged for discussion)

1. Result cache for --update-check resume. This PR fixes the per-invocation rate-limit case (fast-fail then fall through). The bulk --update-check workload still re-scans every package each run; a content-hash result cache (the other half of Resume feature for dealing with rate limits #7) is the real fix and is intentionally out of scope here.
2. Env-var unification. The env-detected chain is kept byte-for-byte (each callX still reads its own legacy env var); the new unified per-field schema applies only to llmN.conf. Unifying env onto that schema — and native Azure/Bedrock support (the api backend is Anthropic-header-only today) — could be a follow-up.
3. Total-failure behaviour. When the whole chain fails, this PR keeps today's fail-closed SUSPICIOUS to stay minimal and avoid touching the gate/hook/score logic. A future option could degrade to the static-rules verdict on total failure (the "rules as last resort" idea), with the automated build hooks kept strict (require INSTALL / fail closed on a degraded scan) while the standalone CLI stays lenient — happy to add this if you'd prefer it.

Open question: in llmN.conf, a backend=/path value is treated as a custom command (mirroring AURSCAN_BACKEND=/path); should a typo'd kind be validated/rejected instead of silently treated as a command path?

🤖 Generated with Claude Code

[manticore-projects]

Thank you much, though I am going to add the following test to internal/scan/chain_test.go (mirrors TestGenuineConfidenceZeroStopsChain):

// TestGenuineSuspiciousStopsChain pins the subtle invariant: a GENUINE
// SUSPICIOUS from an early backend must stop the chain and must NOT be
// overridden by a later backend's OK. (MALICIOUS is covered by
// TestGenuineConfidenceZeroStopsChain; SUSPICIOUS is the trickier case because
// it reads as "fail-ish" and a naive chain might keep trying for an OK.)
func TestGenuineSuspiciousStopsChain(t *testing.T) {
	first, _ := startStub(t, 200, openAIEnvelope(`{"verdict":"SUSPICIOUS","summary":"unverified source"}`))
	second, secondHits := startStub(t, 200, openAIEnvelope(`{"verdict":"OK"}`))
	t.Setenv("AURSCAN_BACKEND", "openai")
	t.Setenv("AURSCAN_OPENAI_URL", first)
	t.Setenv("AURSCAN_OPENAI_MODEL", "")
	setExtraBackends(t, Backend{Kind: "openai", URL: second})

	res := Scan("pkg", testFiles, Signals{})
	if res.V.Verdict != "SUSPICIOUS" || res.Failed {
		t.Fatalf("verdict=%q failed=%v, want SUSPICIOUS/false (genuine SUSPICIOUS must win)", res.V.Verdict, res.Failed)
	}
	if n := atomic.LoadInt32(secondHits); n != 0 {
		t.Fatalf("second backend called %d time(s); a genuine SUSPICIOUS must stop the chain", n)
	}
}

[GeorgelPreput]

Yep, indeed that is something to look out for. Thanks for including the fallback code 🎉