OIDC with JWT fails

[98jan]

Description:
I configured OIDC Authentication with Keycloak and JWT validation (group claim check). The setup works in the beginning as expected, but after a while the Cookie Header get's to big and all requests are blocked.

The cookie get's always to big when the Access Token is expiring after 5min. (default value for keycloak). Then I cann see a lot of cookies with the Name "OauthNonce-" or "CodeVerifier".

The fix for now is to delete the cookies and then I can work for around 5min. again.

Repro steps:

# keycloak crossplane: https://github.com/crossplane-contrib/provider-keycloak
# prometheus client
apiVersion: openidclient.keycloak.crossplane.io/v1alpha1
kind: Client
metadata:
  name: prometheus
spec:
  deletionPolicy: Delete
  forProvider:
    # Enabled standard OAuth2 flow
    standardFlowEnabled: true
    accessType: confidential
    realmId: ${REALM_NAME}
    clientId: prometheus
    rootUrl: "https://prometheus.${DOMAIN}"
    webOrigins:
      - "https://prometheus.${DOMAIN}"
    validRedirectUris:
      - "https://prometheus.${DOMAIN}/oauth2/callback"
    validPostLogoutRedirectUris:
      - "https://prometheus.${DOMAIN}/"
  providerConfigRef:
    name: keycloak-provider-config
  writeConnectionSecretToRef:
    name: prometheus-keycloak-secret
    namespace: monitoring
---
# protocol mapper to add groups to access token
apiVersion: client.keycloak.crossplane.io/v1alpha1
kind: ProtocolMapper
metadata:
  name: prometheus-oidc-group-membership-mapper
spec:
  forProvider:
    name: groups
    realmId: ${REALM_NAME}
    clientIdRef:
      name: prometheus
      #namespace: crossplane
    protocol: openid-connect
    protocolMapper: oidc-group-membership-mapper
    config:
      full.path: "false"
      id.token.claim: "true"
      access.token.claim: "true"
      claim.name: groups
      userinfo.token.claim: "true"
      multivalued: "true"
  providerConfigRef:
    name: keycloak-provider-config
---
# SecurityPolicy for OIDC Authentication
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: prometheus-envoy-oidc
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: kube-prometheus-stack-prometheus
  oidc:
    provider:
      issuer: "https://${KEYCLOAK_DOMAIN}/realms/${REALM_NAME}"
      backendRefs:
        - group: ""
          kind: Service
          name: keycloak-keycloakx-http
          namespace: keycloak
          port: 80
    clientIDRef:
      # transformation is needed, to match key client-id
      name: "keycloak-oidc-client"
    clientSecret:
      # transformation is needed, to match key client-secret
      name: "keycloak-oidc-client"
    redirectURL: "https://prometheus.${DOMAIN}/oauth2/callback"
    # needed that jwt can check claims
    forwardAccessToken: true
    # default is /logout
    #logoutPath: "/logout"
    cookieDomain: "${DOMAIN}"
  jwt:
    providers:
      - name: keycloak
        remoteJWKS:
          failedRefetchDuration: 5s
          uri: https://${KEYCLOAK_DOMAIN}/realms/${REALM_NAME}/protocol/openid-connect/certs
          backendRefs:
            - group: ""
              kind: Service
              name: keycloak-keycloakx-http
              namespace: keycloak
              port: 80
  authorization:
    defaultAction: Deny
    rules:
      - name: "allow-group"
        action: Allow
        principal:
          jwt:
            provider: keycloak
            claims:
              - name: groups
                valueType: StringArray
                values:
                  - prometheus-naj
---
# allow internal communication to keycloak
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-envoy-gateway-to-keycloak
spec:
  from:
    - group: gateway.envoyproxy.io
      kind: SecurityPolicy
      namespace: monitoring
  to:
    - group: ""
      kind: Service
      name: keycloak-keycloakx-http

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Environment:
v1.9.0

[arkodg]

cc @zhaohuabing

[zhaohuabing]

This looks similar to #9632.

Hi @98jan Could you please try if #9644 helps? You'll also be able to configure the TTL of codeVerifier cookie to a shorter time after #9645 lands.

[98jan]

Hi @zhaohuabing ,

what do I need to do to test this feature?
Is this already included in 1.9.0 ?

Best regards,
Jan

[zhaohuabing]

@98jan You can try it with 1.9.1/1.8.4, which will be released soon.

[98jan]

Hi @zhaohuabing ,

just tested it with 1.9.1 and still same behaviour.

[zhaohuabing]

Hi @98jan

Thanks for testing it.

Since Envoy 1.39 each authorization flow gets a random id, and these two cookies are named after it:

OauthNonce-.<flow_id>
CodeVerifier-.<flow_id>

So every redirect to Keycloak creates a new pair under a new name instead of replacing the old one, and Envoy only deletes the pair for the flow that actually completes at /oauth2/callback. Envoy can't delete the cookies for unfinished flows, since cookie deletion can only be done through response headers. My hypothesis is when the access token expires, every XHR in flight gets its own 302 to Keycloak; those are cross-origin and blocked by CORS, so each leaves an orphaned pair behind for its full 600s lifetime. Only the top-level navigation completes and gets cleaned up.

#9644 keeps them off normal requests, but doesn't reduce how many are created — hence no visible change.

Until #9645 lands, bringing both lifetimes under the 5 minute token lifetime should help, so one generation expires before the next arrives:

spec:
  oidc:
    csrfTokenTTL: 290s

and for the code verifier, which has no field yet:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: oidc-code-verifier-ttl
  namespace: monitoring
spec:
  type: JSONPatch
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: <gateway-name>
  jsonPatches:
  - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
    name: <route-config-name>   # <gw-namespace>/<gw-name>/<listener-name>
    operation:
      op: add
      jsonPath: "..typed_per_filter_config['envoy.filters.http.oauth2'].config"
      path: "/code_verifier_token_expires_in"
      value: "290s"

Note these also bound how long there is to complete the login form.

[Preisschild]

My hypothesis is when the access token expires, every XHR in flight gets its own 302 to Keycloak; those are cross-origin and blocked by CORS, so each leaves an orphaned pair behind for its full 600s lifetime

I had this issue with grafana when it created countless 302 cors redirects, which were always blocked by EntraID due to the /authorize page blocking cors requests. Adding

spec:
  oidc:
    denyRedirect:
                headers:
                  - name: Sec-Fetch-Mode
                    type: Exact
                    value: cors

in the SecurityPolicy fixed this issue for me. XHR/fetch requests get 401, no cookies are set.

[98jan]

Thank you very much for sharing the informations.
With the adjustment from @zhaohuabing it got better and with @Preisschild the network tab isn't spammed as much. But when only opening a page after 5min. the page get's unresponsive till I refresh it.

But the header size isn't as big as before.

It's a solution I can work with but still not ideal. Let's see what the feature adjustments will bring.

[Preisschild]

But when only opening a page after 5min. the page get's unresponsive till I refresh it.

It seems SPAs have to handle fetch/xhrs failing with 401 and then doing a top level navigation request so you get redirected to your OIDC IDP /authorize page

Or you use refresh Tokens with the offline_access scope, those can be renewed via fetch/xhr