Skip to content

fix: fetch the OIDC discovery document lazily instead of at import time - #133

Open
bladewing wants to merge 1 commit into
siemens:masterfrom
bladewing:fix/lazy-well-known-discovery
Open

fix: fetch the OIDC discovery document lazily instead of at import time#133
bladewing wants to merge 1 commit into
siemens:masterfrom
bladewing:fix/lazy-well-known-discovery

Conversation

@bladewing

@bladewing bladewing commented Aug 26, 2026

Copy link
Copy Markdown

Problem

oidc/constants.py performs a requests.get() against <OIDC_DOMAIN>/.well-known/openid-configuration with timeout=2.0 while the module is being imported. Since the provider is registered from apps.py, every Sentry process imports it — including the ~20 consumer/worker containers of a self-hosted installation that never perform a login.

Whenever the identity provider takes longer than 2 s to answer, every one of those processes dies with

  File "/.venv/lib/python3.13/site-packages/oidc/constants.py", line 20, in <module>
    well_known_values = requests.get(WELL_KNOWN_URL, timeout=2.0).json()
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='auth.example.com', port=443): Read timed out. (read timeout=2.0)

and is restarted immediately by Docker (restart: unless-stopped). Twenty Python processes booting at once drive up the load, the IdP gets slower, and the loop feeds itself. On our installation this ran for months at ~189,000 restarts per consumer and yesterday reached a load average of ~1,300 that made the host unreachable (sshd and the reverse proxy stopped answering).

Fix

  • The discovery document is fetched on first use — when a login pipeline is built (OIDCLogin, OAuth2Callback, get_refresh_token_url, get_user_info) or the configure view is rendered — via get_well_known(), and cached for the lifetime of the process.
  • A failed fetch is logged as a warning and yields {}; the accessors fall back to the statically configured OIDC_*_ENDPOINT settings instead of raising. After a failure the provider is not contacted again for OIDC_WELL_KNOWN_RETRY_AFTER seconds (default 30) so a slow IdP cannot stall every request.
  • Timeout raised from 2 s to 15 s and made configurable via OIDC_WELL_KNOWN_TIMEOUT.
  • Module-level constants AUTHORIZATION_ENDPOINT, TOKEN_ENDPOINT, USERINFO_ENDPOINT, ISSUER are kept (now purely settings-derived) so existing imports keep working; new accessors get_authorization_endpoint(), get_token_endpoint(), get_userinfo_endpoint(), get_issuer(), get_provider_name() return the discovered values.

Behaviour change to be aware of

PROVIDER_NAME (used as the class attribute OIDCProvider.name) no longer falls back to the discovered issuer, because a class attribute is evaluated at import time and must not depend on network I/O. It still honours OIDC_PROVIDER_NAME, then OIDC_ISSUER, then "OIDC". The configure view keeps the discovered issuer as a fallback through get_provider_name(). README updated accordingly. If you would rather keep the old fallback for the class attribute, I'm happy to discuss options (e.g. a metaclass property), but I think a static name is the safer default.

Testing

  • New tests/test_discovery_unit.py (uses the existing Sentry stubs from test_provider_unit.py): import performs no request; first pipeline build fetches once and caches; a ReadTimeout does not raise and falls back to static settings, with the back-off honoured; configured OIDC_ISSUER takes precedence; without OIDC_DOMAIN nothing is fetched.
  • ruff check / ruff format --check pass.
  • An equivalent patch (against 9.1.0) has been running on our production self-hosted Sentry since yesterday: restart counts stayed at 0, load normalised, and SSO login via Authentik still redirects correctly to the discovered authorization_endpoint.

@CLAassistant

CLAassistant commented Aug 26, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@bladewing
bladewing force-pushed the fix/lazy-well-known-discovery branch from c2b7dca to ba86698 Compare August 26, 2026 07:58
`oidc/constants.py` performed a `requests.get()` against
`<OIDC_DOMAIN>/.well-known/openid-configuration` with a 2 second timeout
while the module was being imported. Every Sentry process imports the
provider through `apps.py`, including the ~20 consumer and worker
containers of a self-hosted installation that never perform a login.
Whenever the identity provider took longer than 2 seconds to answer, all
of those processes died with `requests.exceptions.ReadTimeout` during
import and were restarted immediately by Docker, which in turn put more
load on the host (and on the identity provider if it runs on the same
machine). On one installation this resulted in ~189,000 restarts per
consumer and a load average of ~1,300 that made the host unreachable.

The discovery document is now fetched on first use (when a login
pipeline is built or the configure view is rendered), cached for the
lifetime of the process, and a failed fetch is logged as a warning and
falls back to the statically configured `OIDC_*_ENDPOINT` settings
instead of raising. The timeout was raised to 15 seconds and both the
timeout and the back-off after a failure are configurable through
`OIDC_WELL_KNOWN_TIMEOUT` and `OIDC_WELL_KNOWN_RETRY_AFTER`.

`PROVIDER_NAME` (used as the class-level `OIDCProvider.name`) no longer
falls back to the *discovered* issuer, because a class attribute cannot
depend on network I/O; it still honours `OIDC_PROVIDER_NAME` and
`OIDC_ISSUER`. The configure view keeps the discovered issuer as a
fallback via `get_provider_name()`.
@bladewing
bladewing force-pushed the fix/lazy-well-known-discovery branch from ba86698 to 767350e Compare August 27, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants