User part of multiple groups in keycloak is denied access to Jupyterhub

[mt-cpo]

Bug description

I am installing jupyterhub using Helm and Keycloak for authentication. When a user is part of one group (ex: jupyter_users) and I declared in:

allowed_groups:
        - "/jupyter_users"

It works, but if I add that same user to another group (ex: he is part of jupyter_users and foo group in keycloak) then he get denied access to jupyterhub.

How to reproduce

1. Helm install using the following config:

hub:
  config:
    JupyterHub:
        authenticator_class: generic-oauth
    GenericOAuthenticator:
        login_service: "Keycloak"
        client_id: "$JUPYTERHUB_CLIENT_ID"
        client_secret: "$client_secret"
        oauth_callback_url: "http://$JUPYTERHUB_DOMAIN/hub/oauth_callback"
        authorize_url: "http://$KEYCLOAK_DOMAIN/realms/$KEYCLOAK_REALM/protocol/openid-connect/auth"
        token_url: "http://$KEYCLOAK_DOMAIN/realms/$KEYCLOAK_REALM/protocol/openid-connect/token"
        userdata_url: "http://$KEYCLOAK_DOMAIN/realms/$KEYCLOAK_REALM/protocol/openid-connect/userinfo"
        scope:
        - openid
        - email
        - profile
        - groups
        username_claim: "preferred_username"
        allowed_groups:
        - "/jupyterhub_users"
        admin_groups:
        - "/jupyterhub_admins"
        claim_groups_key: "groups"

singleuser:
    defaultUrl: "/lab"

proxy:
  service:
    type: $SERVICE_TYPE
    nodePorts:
      http: 30696
      https: 30696
  https:
    enabled: true

2. Create a group in Keycloak called jupyterhub_users
3. Create a user and add him to group jupyterhub_users
4. Login to Jupyterhub -> This should work
5. Create a second group in Keycloak called foo
6. Add the same user to the group foo
7. Login to Jupyterhub -> This should throw 500 internal server error

Logs

hub-55f66fc556-g6522 [D 2024-09-17 10:58:43.282 JupyterHub reflector:374] pods watcher timeout
hub-55f66fc556-g6522 [D 2024-09-17 10:58:43.282 JupyterHub reflector:289] Connecting pods watcher
hub-55f66fc556-g6522 [D 2024-09-17 10:58:44.152 JupyterHub log:192] 200 GET /hub/health (@10.244.0.1) 1.29ms
hub-55f66fc556-g6522 [D 2024-09-17 10:58:46.151 JupyterHub log:192] 200 GET /hub/health (@10.244.0.1) 0.83ms
hub-55f66fc556-g6522 [E 2024-09-17 10:58:47.459 JupyterHub oauth2:683] Error Fetching user info... 401 GET http://keycloak.default/realms/omniops/protocol/openid-connect/userinfo: 
hub-55f66fc556-g6522 [E 2024-09-17 10:58:47.460 JupyterHub web:1875] Uncaught exception GET /hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJmZTg1ZWUzODdhZmM0ZDNmOGQxMmVmNWQ4ZDAxMDNmMCJ9&session_state=647de1bc-b9dc-4da9-99a1-ba03190c01ba&iss=http%3A%2F%2Fkeycloak.default%2Frealms%2Fomniops&code=dd3f8514-8d4a-4b76-9b50-065c84135236.647de1bc-b9dc-4da9-99a1-ba03190c01ba.a481311f-ef17-4bb2-8452-8e0566214afb (::ffff:10.244.0.1)
hub-55f66fc556-g6522     HTTPServerRequest(protocol='http', host='jupty', method='GET', uri='/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJmZTg1ZWUzODdhZmM0ZDNmOGQxMmVmNWQ4ZDAxMDNmMCJ9&session_state=647de1bc-b9dc-4da9-99a1-ba03190c01ba&iss=http%3A%2F%2Fkeycloak.default%2Frealms%2Fomniops&code=dd3f8514-8d4a-4b76-9b50-065c84135236.647de1bc-b9dc-4da9-99a1-ba03190c01ba.a481311f-ef17-4bb2-8452-8e0566214afb', version='HTTP/1.1', remote_ip='::ffff:10.244.0.1')
hub-55f66fc556-g6522     Traceback (most recent call last):
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1790, in _execute
hub-55f66fc556-g6522         result = await result
hub-55f66fc556-g6522                  ^^^^^^^^^^^^
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 210, in get
hub-55f66fc556-g6522         user = await self.login_user()
hub-55f66fc556-g6522                ^^^^^^^^^^^^^^^^^^^^^^^
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 928, in login_user
hub-55f66fc556-g6522         authenticated = await self.authenticate(data)
hub-55f66fc556-g6522                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/jupyterhub/auth.py", line 493, in get_authenticated_user
hub-55f66fc556-g6522         authenticated = await maybe_future(self.authenticate(handler, data))
hub-55f66fc556-g6522                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 1063, in authenticate
hub-55f66fc556-g6522         user_info = await self.token_to_user(token_info)
hub-55f66fc556-g6522                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 978, in token_to_user
hub-55f66fc556-g6522         return await self.httpfetch(
hub-55f66fc556-g6522                ^^^^^^^^^^^^^^^^^^^^^
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 718, in httpfetch
hub-55f66fc556-g6522         return await self.fetch(
hub-55f66fc556-g6522                ^^^^^^^^^^^^^^^^^
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 684, in fetch
hub-55f66fc556-g6522         raise e
hub-55f66fc556-g6522       File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 663, in fetch
hub-55f66fc556-g6522         resp = await self.http_client.fetch(req, **kwargs)
hub-55f66fc556-g6522                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hub-55f66fc556-g6522     tornado.httpclient.HTTPClientError: HTTP 401: Unauthorized
hub-55f66fc556-g6522     
hub-55f66fc556-g6522 [D 2024-09-17 10:58:47.461 JupyterHub base:1471] No template for 500
hub-55f66fc556-g6522 [E 2024-09-17 10:58:47.475 JupyterHub log:184] {
hub-55f66fc556-g6522       "X-Forwarded-Host": "jupty",
hub-55f66fc556-g6522       "X-Forwarded-Proto": "http",
hub-55f66fc556-g6522       "X-Forwarded-Port": "80",
hub-55f66fc556-g6522       "X-Forwarded-For": "::ffff:10.244.0.1",
hub-55f66fc556-g6522       "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
hub-55f66fc556-g6522       "Accept-Language": "en-US,en;q=0.9",
hub-55f66fc556-g6522       "Accept-Encoding": "gzip, deflate",
hub-55f66fc556-g6522       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
hub-55f66fc556-g6522       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
hub-55f66fc556-g6522       "Upgrade-Insecure-Requests": "1",
hub-55f66fc556-g6522       "Cache-Control": "max-age=0",
hub-55f66fc556-g6522       "Connection": "keep-alive",
hub-55f66fc556-g6522       "Host": "jupty"
hub-55f66fc556-g6522     }
hub-55f66fc556-g6522 [E 2024-09-17 10:58:47.475 JupyterHub log:192] 500 GET /hub/oauth_callback?state=[secret]&session_state=[secret]&iss=http%3A%2F%2Fkeycloak.default%2Frealms%2Fomniops&code=[secret] (@::ffff:10.244.0.1) 27.21ms

[consideRatio]

This seems like a config issue because of the leading /

        allowed_groups:
        - "/jupyterhub_users"
        admin_groups:
        - "/jupyterhub_admins"

[mt-cpo]

thanks @consideRatio for the reply. No, it works with the / because that's what Keycloak is sending. If the / was the problem it wouldn't work when the user is part of one group only.

We found a workaround which works now, but not sure if this is best practice:

extraConfig: 
    00-custom-authenticator: |
      from oauthenticator.generic import GenericOAuthenticator

      class CustomAuthenticator(GenericOAuthenticator):
          allowed_group = '/jupyter_users'  # Specify your allowed group
          admin_group = '/jupyter_admin'  # Specify your admin group

          async def authenticate(self, handler, data):
              user_info = await super().authenticate(handler, data)
              if user_info:
                  # Get the groups from the token
                  groups = user_info.get('auth_state', {}).get('oauth_user', {}).get('groups', [])
                  print(f"---------{groups}")
                  # Check if the user belongs to the allowed group
                  if self.allowed_group in groups or self.admin_group:
                      return user_info  # Allow login if in allowed group
                  else:
                      return None  # Deny access if not in allowed group

      c.JupyterHub.authenticator_class = CustomAuthenticator
      c.GenericOAuthenticator.scope = ['openid', 'profile', 'email', 'groups']