fix: stop trusting a caller-supplied X-Forwarded-For (#210) - #213
Open
mekarpeles wants to merge 3 commits into
Open
fix: stop trusting a caller-supplied X-Forwarded-For (#210)#213mekarpeles wants to merge 3 commits into
mekarpeles wants to merge 3 commits into
Conversation
LENNY_FORWARDED_ALLOW_IPS defaulted to '*'. Lenny's nginx sets the header with
$proxy_add_x_forwarded_for, which APPENDS the address it observed to whatever
the client already sent, and uvicorn's ProxyHeadersMiddleware returns the
LEFTMOST entry when it trusts everything. So a caller could prepend any value
and choose the IP Lenny binds session cookies and OTPs to, making every
IP-based check advisory rather than enforced.
Three sites carried the wildcard, not one:
- docker/api/Dockerfile CMD — the live path. configs.OPTIONS only reaches
uvicorn under `python -m lenny.app`, so this is what production actually ran.
- lenny/configs/__init__.py — the `python -m lenny.app` path.
- docker/configure.sh — did not template the variable at all, so fresh installs
inherited whichever fallback applied.
Bounded, uvicorn instead walks the chain right-to-left and returns the first
untrusted hop: the address nginx actually observed.
configure.sh now resolves the real Compose subnet when it can, falling back to
172.16.0.0/12. The lookup is deliberately kept OUT of the `${VAR:-default}`
expression: docker/utils/update/020_env_sync.sh back-fills new keys into
existing installs by grepping this file for that literal form and resolves a
`$(...)` default to the EMPTY STRING. Inline, it would have written
`LENNY_FORWARDED_ALLOW_IPS=` into every existing .env on the next update — and
empty is worse than '*', because uvicorn then trusts no proxy at all and every
patron collapses onto the nginx address: one identity, one shared OTP
rate-limit bucket (OL allows 3/60s per ip), and a session binding that matches
from anywhere. Verified by running env_sync's own extraction pipeline against
the new file: it resolves to 172.16.0.0/12, not empty.
Both failure modes were completely silent, which is why this survived from the
day proxy headers were enabled. Added a startup warning when the range is '*',
and a once-per-worker warning on the first request whose resolved client.host
is not the last hop of the chain — which catches too-broad and too-narrow with
one check.
14 new tests driving uvicorn's real ProxyHeadersMiddleware rather than
asserting on our config string: a spoofed chain must lose to the observed
address, the nginx $proxy_add_x_forwarded_for shape must still yield the real
patron IP (the #201 regression), a blank env var must fall back rather than
disable proxy trust, and neither shipped entrypoint may carry a wildcard.
Mutation-checked: restoring the '*' default fails 7. Full suite 307 passed,
flake8 clean, and the shell guard exercised against empty, error, and
whitespace lookups — it never emits an empty value.
Prod was already fixed by hand (172.18.0.0/16); this fixes the defaults every
other install inherits.
Review points from olsystem-3b, both verified rather than argued. Ordering: uvicorn.config.Config.load wraps the loaded app as ProxyHeadersMiddleware(loaded_app, trusted_hosts=...), i.e. OUTSIDE every Starlette/FastAPI middleware, so the warning observes an already-resolved scope['client']. Six tests drive the real stack to pin that, rather than trusting it — if uvicorn ever moved the wrap inside, the warning would silently start reading the raw peer and fire on every request. Building them surfaced a trap: TestClient's peer is the literal string 'testclient', which is in no trusted range, so ProxyHeadersMiddleware ignored the header entirely and the first version of the test measured nothing. The peer is now rewritten by a thin ASGI wrapper outside the middleware. That also gave a test worth keeping on its own: a caller reaching uvicorn directly, not via nginx, has its header ignored outright. Warn-once: the flag is module-level, so one warning per worker process and up to three at the default --workers=3. Intended (it is a startup-class diagnostic), now covered by a test asserting exactly one warning across five requests, so a reviewer does not read it as a per-request bug. 313 passed, flake8 clean.
"Middleware ordering" names two unrelated things here, and the comment did not distinguish them. uvicorn applies ProxyHeadersMiddleware in Config.load, wrapping the entire Starlette stack from outside, so scope['client'] is resolved before any app middleware runs and this warning's position among them is irrelevant. Add-order only decides nesting among Starlette's own middleware — which is what makes something like a CORS override need to be added last to wrap outermost. Also records that one warning per worker means up to three at the default --workers=3, and that this is intended. Comment only; 313 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #210. Prod was already patched by hand (
172.18.0.0/16); this fixes the defaults every other install inherits, including fresh ones.The bug
LENNY_FORWARDED_ALLOW_IPSdefaulted to*. nginx sets the header with$proxy_add_x_forwarded_for, which appends the address it observed to whatever the client already sent:With
always_trust, uvicorn returnsx_forwarded_for_hosts[0]— the leftmost, i.e. the client's own value. So a caller could choose the IP Lenny binds session cookies and OTPs to. Every IP check became advisory.Bounded, uvicorn instead walks right-to-left and returns the first untrusted hop: the address nginx actually saw.
Three sites, not one
docker/api/DockerfileCMDconfigs.OPTIONSonly reaches uvicorn underpython -m lenny.app, so this is what production ranlenny/configs/__init__.py:76python -m lenny.apppathdocker/configure.shWhy the subnet lookup is not inline
configure.shnow resolves the real Compose subnet when it can, falling back to172.16.0.0/12. The lookup is deliberately kept out of the${VAR:-default}expression.docker/utils/update/020_env_sync.shback-fills new keys into existing installs by grepping this file for the literalKEY="${KEY:-<default>}"form, and explicitly resolves a$(...)default to the empty string (020_env_sync.sh:163-170). Inline, it would have writtenLENNY_FORWARDED_ALLOW_IPS=into every existing.envon the nextmake update.Empty is worse than
*. uvicorn then trusts no proxy at all, so every patron collapses onto the nginx address — one identity, one shared OTP rate-limit bucket (OL allows 3/60s per ip), and a session binding that matches from anywhere. That's the outage-class bug #201 existed to fix.Verified by running env_sync's own extraction pipeline against the new file:
And the shell guard against every failure mode:
Making it loud
Both failure modes are silent — no exception, no status change, no log line. That is why this survived from the day proxy headers were enabled. Added:
*;client.hostis not the last hop of the chain. One check catches both directions: matching the first hop means too broad (spoofable), matching neither means too narrow (everyone collapsed).Testing
ProxyHeadersMiddleware, not asserting on our config string — so they fail for the same reason production would.$proxy_add_x_forwarded_forshape still yields the real patron IP (the fix: surface OTP failures instead of swallowing them #201 regression, previously untested); a blank env var falls back rather than disabling proxy trust; an operator override is honoured; neither shipped entrypoint carries a wildcard;configure.shtemplates the key.*behaviour so the regression stays legible.*default fails 7 of them.bash -nclean.Coordination
Found and scoped with
olsystem-3b, who filed #210 and fixed prod. Their #211 (draft) also touchesdocker/configure.sh's.envheredoc — my addition is contiguous at the end and should rebase either direction; they've said they'll take the conflict and move #211 to0.2.19if this lands first. I've leftnginx.conf,Makefile,models.py, and everything underoauth2untouched.Not verified against a running Lenny — the middleware behaviour is exercised directly, and prod already carries an equivalent value set by hand.