[ENH] KC Admin API URL should have a backchannel override, like token/userinfo do

[oren-openteams]

Problem

Chart 0.1.3 supports OIDC split-horizon via keycloak.backchannelURL: token_url and userdata_url are derived from this in-cluster URL while authorize_url and end_session_url continue to use the external keycloak.hostname. This makes the hub Pod on private-VPC / split-DNS clusters work correctly for the browser-facing legs and the hub-facing token/userinfo legs.

But KeyCloakOAuthenticator.realm_api_url (the KC Admin API URL used by the chart's own RBAC filter — filter_user_groups_by_role, get_profile_slugs_for_user) is derived from keycloak.hostname too, with no equivalent backchannel override. On clusters where the hub Pod cannot resolve the external Keycloak hostname (common on private-VPC deployments with split DNS), the Admin API calls fail:

[E ... oauth2:857] Error fetching 599 GET https://keycloak.<domain>/admin/realms/<realm>/clients:
    HTTP 599: Could not resolve host: keycloak.<domain>
[W ... 00-gateway-auth:414] rbac: failed to refresh groups_with_permission_to_mount for <user> —
    keeping last known set

Because the chart's exception handler in _get_user_groups preserves the previous (usually empty) auth_state["groups_with_permission_to_mount"] on error, the failure is silent: RBAC returns nothing, and every user's /shared/<group> mounts vanish. There is no visible spawn error — just missing folders. On the affected cluster this took several hours to root-cause, because the surface symptom (no shared mounts) points at spawner or auth_state hooks rather than an Admin API DNS failure.

Setting realm_api_url via hub.extraConfig to http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080/admin/realms/<realm> fixes it immediately — the Admin API uses a bearer-token flow (obtained via oauthenticator's client-credentials grant), so pointing at the in-cluster Service address has no iss-claim reconciliation issue like the OIDC token leg had.

Proposed enhancement

Add a chart value that drives realm_api_url the same way keycloak.backchannelURL drives token_url and userdata_url. Two shapes come to mind:

Option A — new dedicated value

keycloak:
  hostname: keycloak.example.com          # browser-facing
  backchannelURL: http://keycloak.svc.cluster.local:8080   # already exists; drives token/userinfo
  adminBackchannelURL: ""                # NEW; drives realm_api_url. Empty = derived from hostname.

Option B — extend the existing backchannelURL to cover the admin API leg

When keycloak.backchannelURL is set, ALSO derive realm_api_url from it. Rationale: any cluster that needs a backchannel URL for token/userinfo (because external hostname doesn't resolve in-cluster) needs one for the Admin API for the same reason.

I'd advocate for Option B — I can't come up with a realistic scenario where a deployer would want the token backchannel to be in-cluster but the Admin API leg to keep failing on the external hostname. Simpler surface; one less value.

Either way, docstring should call out this failure mode explicitly ("silent RBAC failure → /shared/* mounts missing → looks like a spawner bug") because it's easy to miss.

Reproduction

• Deploy chart 0.1.3 with keycloak.hostname: keycloak.example.com and keycloak.backchannelURL: http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080 on any cluster where the hub Pod can't resolve keycloak.example.com.

• Ensure KeyCloakOAuthenticator is the runtime authenticator (chart default), realm_api_url is set (chart derives it from hostname), and at least one Keycloak group holds the allow-group-directory-creation-role client role on the hub client.

• Have that user log in fresh, decrypt their auth_state:

  groups_with_permission_to_mount: []       # RBAC failure preserved as empty
  auth_state.groups             : [...]     # populated correctly
  oauth_user.groups             : [...]     # populated correctly
  

• Spawn a user pod: no /shared/* mounts, no initialize-shared-mounts init container.

• Hub logs show HTTP 599: Could not resolve host for .../admin/realms/<realm>/clients.

Workaround

Override realm_api_url in hub.extraConfig alongside the existing 99-oauth-style entries:

c.KeyCloakOAuthenticator.realm_api_url = "http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080/admin/realms/nebari"

Also verified: silent RBAC failure preserves the last known groups_with_permission_to_mount value; on FIRST auth after a broken chart install that value is [], so users get an empty allow-list and no mounts. Consider logging at a higher level (INFO or WARN) when the fetch throws — even one loud message on install would have cut root-cause time in half here.

Related

• Chart's KC subclass in config/jupyterhub/00-gateway-auth.py (see KeyCloakConfig.build() + the wire_keycloak call). KeyCloakConfig.build() already accepts a backchannel_issuer and derives token_url/userdata_url from it — an equivalent could easily derive realm_api_url from the same input.
• sharedStorage.groups allowlist rendered in values but not consumed by chart templates #206 (sharedStorage.groups allowlist) — different bug but adjacent surface area; both surface as silent shared-storage misbehavior.