Give a route one optional fallback upstream, via a metadata-only ordered plan

[lgoyal6]

What this changes

A route may now name one optional fallback upstream. The router turns the
row into an ordered route plan - primary, then at most one fallback - and the
proxy walks it.

This is an independently designed extension of this project's existing route,
retry and credential-injection model.

Router versus proxy

The plan is routing metadata and nothing else: which upstreams, in what
order, and the name of the environment variable each one's credential comes
from. It never holds, reads or borrows a request byte, and never holds a secret.

The proxy stays the only component allowed to read or buffer a request body,
inject a provider credential, open an upstream connection, retry, hedge, fail
over, stream a response, or decide whether bytes are replayable.

That split is why the candidate order can be logged and traced without a body or
a credential going near a log line.

One fallback, not a routing policy

No scoring, no weighting, no plugin point, no arbitrary-length candidate graph,
no policy registry, no feature flag. One optional fallback is the complete
scope: enough to survive a single upstream refusing service, small enough that
the order reads off one database row.

Exact failover eligibility

Failover happens only when all of these hold:

• the method is idempotent (GET/HEAD/OPTIONS) and the body was buffered;
• the client context is live and the request is inside its total deadline;
• no response bytes have been written to the client yet;
• an attempt remains inside the route's existing ceiling;
• and the primary failed with a transport error, an open circuit breaker, or a
  502, 503 or 504.

It does not happen for: POST, an unbuffered or over-limit body, 401, 403,
429 or any other 4xx, a partially streamed response, a cancelled client, or a
credential the gateway does not have. Hedging is unchanged and races the primary
only.

POST is the one worth stating plainly: an LLM completion may already have been
generated and billed upstream, and "it returned 503" is not evidence that it did
nothing.

Shared attempt and budget semantics

A fallback is an attempt spent out of the route's existing 1 + retry_max
ceiling, not a second retry budget. One attempt of that ceiling is reserved
for the fallback so it can actually be reached - which is why a fallback
requires retries >= 1, and why configuring one without it is refused rather
than stored as config that silently never fires.

Budget is untouched: one spend hold and one settlement per request, however
many upstreams it touched. Tested directly.

Configuration

Smallest backward-compatible representation: four nullable-by-default columns on
routes (migration 005). Every existing row gets '', plans a single
candidate, and behaves exactly as before. No backfill, no operator action, no
migration window. The existing statement-level routes_notify trigger already
covers the new columns.

tollgate-admin add-route -tenant acme -prefix /api/ -upstream http://upstream-a:9000 \
    -retries 1 -fallback http://upstream-b:9000 \
    -fallback-auth-header x-api-key -fallback-auth-env BACKUP_API_KEY

tollgate-admin set-fallback   -route 3 -upstream http://upstream-b:9000
tollgate-admin clear-fallback -route 3

The fallback names its own header and environment variable. A fallback is
usually a different provider, and one shared env var is how one provider's key
reaches another one.

Four configurations are refused rather than stored: a fallback on a route with
no retries, a fallback equal to the primary, a fallback the upstream allowlist
refuses, and fallback credentials with no fallback upstream. The router also
runs the upstream allowlist check over every candidate - a fallback pointed
at the metadata service is the same SSRF bug with one more step in front of it.

Negative controls

All four are tests, and each was confirmed to fail when the corresponding guard
was deliberately removed:

Control	Expected	Result
primary consumes the body, then returns 503	fallback receives the byte-identical body	passes; fails when the fallback is sent a different reader
POST receives 503	fallback receives zero requests	passes; fails when both the ceiling and the eligibility guard are removed
unknown-length, non-replayable request fails	fallback receives zero requests	passes; fails under the same pair of mutations
response begins streaming, then fails	fallback receives zero requests	passes; fails when a served response is made abandonable

POST and the unbuffered body turned out to be defended twice over - once by the
attempt ceiling (attemptCeiling returns 1 when a request cannot be repeated)
and once by the eligibility check. Removing either alone leaves the controls
green; removing both turns them red. That is defence in depth, and it is
recorded rather than presented as one guard.

Also covered: 401/403/429 never fail over, a cancelled client never starts a
fallback, the fallback uses only its own credential source, the primary's
credential never reaches the fallback, no caller credential reaches either
upstream, a missing fallback credential does not send an unauthenticated
request, both candidates see the same path, and the log names the candidate
order and chosen candidate while leaking neither credential nor body.

Test commands

go test ./internal/proxy ./internal/middleware ./internal/store ./internal/admin -count=1
go test ./... -count=1
go test ./internal/proxy -run 'TestOrderedRoutePlanEvaluation' -count=1 -v
git diff --check

All pass. go vet ./... clean, gofmt -l . empty, and
go test -race ./internal/proxy ./internal/middleware -count=1 passes.

The entire pre-existing suite passes unchanged, which is the
"single-upstream routes behave exactly as before" invariant.

Evaluation

go test ./internal/proxy -run TestOrderedRoutePlanEvaluation -v, 300 requests
per arm, artifact at results/route-plan-eval.json:

arm	success	attempts/req	fallbacks	p50	p95	p99	body-integrity failures
failing primary, single upstream	0.00%	2.00	0	14.756ms	25.143ms	26.273ms	0
failing primary, ordered plan	100.00%	2.00	300	0.151ms	0.335ms	0.486ms	0
all healthy, single upstream	100.00%	1.00	0	0.041ms	0.061ms	0.183ms	0
all healthy, ordered plan	100.00%	1.00	0	0.040ms	0.053ms	0.106ms	0

With a healthy primary the fallback is contacted zero times and the request
costs the same one attempt, so there is no measurable overhead when it is not
needed. The latency gap in the failing case is the retry backoff, not
cleverness: the single-upstream arm waits and asks the same dead upstream again,
while the plan goes straight to a different one with no backoff between
candidates.

The test asserts on the deterministic properties and merely reports the numbers.
A latency threshold in a unit test is a flake with a date on it.

Limitations

• Two httptest servers and a proxy, in one process, on one machine. The
  "outage" is a handler that was told to return 503.
• Not a datacenter, a provider, a network partition, or a real failover. The
  success rates are arithmetic about a scripted condition, not evidence about
  availability.
• Says nothing about LLM traffic, which is POST and never fails over.
• Latency is loopback wall-clock: measured, not deterministic.
• The breaker is configured not to open inside an evaluation run, so the plan is
  what is being measured. The breaker-open failover path has its own unit test.
• Hedging and failover are not combined. Hedging is already two in-flight
  attempts at one upstream, and sharing one ceiling between that and a second
  upstream is a larger change than one fallback.

🤖 Generated with Claude Code