cull_idle not being able to clean up because of HTTP: 403 Forbidden

[hnykda]

Bug description

Even though my chart has values:

cull:
  enabled: true

user hubs are not cleaned up. I am on chart v4.0.0, azure deployment. The problem is that culler can't seem to get to the hub API:

│ [I 2025-01-20 20:15:13.029 JupyterHub log:192] 200 POST /hub/api/users/jackwildman/activity (jackwildman@10.244.0.121) 15.16ms                                                                                                                                                                                                                                                   │
│ [I 2025-01-20 20:16:20.171 JupyterHub log:192] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 10.61ms                                                                                                                                                                                                                                                                            │
│ [W 2025-01-20 20:16:20.174 JupyterHub scopes:1019] Not authorizing access to /hub/api/users. Requires any of [list:users] on *, not derived from scopes []                                                                                                                                                                                                                       │
│ [W 2025-01-20 20:16:20.174 JupyterHub web:1873] 403 GET /hub/api/users?state=ready (::1): Action is not authorized with current scopes; requires any of [list:users]
│ [W 2025-01-20 20:16:20.174 JupyterHub log:192] 403 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 1.65ms                                                                                                                                                                                                                                                         │
│ [E 250120 20:16:20 ioloop:941] Exception in callback functools.partial(<function cull_idle at 0x7f41a330b6a0>, url='http://localhost:8081/hub/api', api_token='89f1f94b53c643bb8abbbbb4f9b27180bd87ead', inactive_limit=3600, cull_users=False, remove_named_servers=False, max_age=0, concurrency=10, ssl_enabled=False, internal_certs_location='internal-ssl', cull_admin_users=True │
│ , api_page_size=0, cull_default_servers=True, cull_named_servers=True)                                                                                                                                                                                                                                                                                                           │
│     Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                           │
│       File "/usr/local/lib/python3.12/site-packages/tornado/ioloop.py", line 939, in _run                                                                                                                                                                                                                                                                                        │
│         await val                                                                                                                                                                                                                                                                                                                                                                │
│       File "/usr/local/lib/python3.12/site-packages/jupyterhub_idle_culler/__init__.py", line 436, in cull_idle                                                                                                                                                                                                                                                                  │
│         async for user in fetch_paginated(req):                                                                                                                                                                                                                                                                                                                                  │
│       File "/usr/local/lib/python3.12/site-packages/jupyterhub_idle_culler/__init__.py", line 142, in fetch_paginated                                                                                                                                                                                                                                                            │
│         response = await resp_future                                                                                                                                                                                                                                                                                                                                             │
│                    ^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                             │
│       File "/usr/local/lib/python3.12/site-packages/jupyterhub_idle_culler/__init__.py", line 124, in fetch                                                                                                                                                                                                                                                                      │
│         return await client.fetch(req)                                                                                                                                                                                                                                                                                                                                           │
│                ^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                           │
│     tornado.httpclient.HTTPClientError: HTTP 403: Forbidden                                                                                                                                                                                                                                                                                                                      │
│ [I 2025-01-20 20:16:33.478 JupyterHub log:192] 200 GET /hub/api/user (dschwarz26@10.244.0.135) 10.44ms

[manics]

Can you share your full configuration?

[hnykda]

Sure:

# This file can update the JupyterHub Helm chart's default configuration values.
#
# For reference see the configuration reference and default values, but make
# sure to refer to the Helm chart version of interest to you!
#
# Introduction to YAML:     https://www.youtube.com/watch?v=cdLNKUoMc6c
# Chart config reference:   https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html
# Chart default values:     https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/HEAD/jupyterhub/values.yaml
# Available chart versions: https://hub.jupyter.org/helm-chart/
#

proxy:
  # we don't need a separate loadbalancer, we will use Traefik
  service:
    type: ClusterIP
    disableHttpPort: false
  https:
    enabled: false

ingress:
  enabled: false

hub:
  config:
    Authenticator:
      admin_users:
        - ****
    GitHubOAuthenticator:
      client_id: ****
      client_secret: ****
      oauth_callback_url: https://****/hub/oauth_callback
      allowed_organizations:
        - ****
      enable_auth_state: true
      allow_all: true
    JupyterHub:
      authenticator_class: github
      load_roles:
        - name: user
          description: Ovewriting default to be able to share access to your servers
          scopes:
            - self
            # as per https://jupyterhub.readthedocs.io/en/stable/reference/sharing.html
            - shares!user
            - read:users:name
            - read:groups:name
            # these are for being able to access the admin ui https://discourse.jupyter.org/t/real-time-collaboration-z2jh/24059/6
            # could be separated in their own groups
            - access:servers
            - admin:servers
            - admin-ui
            - list:users
singleuser:
  # Defines the default image
  image:
    name: us-central1-docker.pkg.dev/***/jupyterhub-user-image/jupyter-alpha
    tag: "0.0.6"
    pullSecrets:
      - name: gcp-us-central1-secret
  profileList:
    - display_name: "Default Python Data Science Env"
      description: "**** custom data analysis stack with some extra extensions"
      default: true
      kubespawner_override:
        lifecycle_hooks:
          postStart:
            exec:
              command:
                - "sh"
                - "-c"
                - >
                  cp /tmp/StartHere.ipynb /home/jovyan/ &&
                  cp -r /tmp/.condarc /home/jovyan/.condarc &&
                  cp -r /tmp/.ssh /home/jovyan/.ssh &&
                  chmod 600 /home/jovyan/.ssh/id_rsa &&
                  [ ! -d /home/jovyan/research ] && cp -r /tmp/research /home/jovyan/research || true
    - display_name: "Minimal default scipy notebook"
      description: "Minimal default scipy notebook environment"
      kubespawner_override:
        image: quay.io/jupyter/scipy-notebook
        tag: latest

  extraEnv:
    EDITOR: "vim"
    SERPER_API_KEY:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: SERPER_API_KEY
    ANTHROPIC_API_KEY:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: ANTHROPIC_API_KEY
    OPENAI_API_KEY:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: OPENAI_API_KEY
    RETROSEARCH_URL:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: RETROSEARCH_URL
    GEMINI_API_KEY:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: GEMINI_API_KEY
    DAGSTER_HOME:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: DAGSTER_HOME
    PYTHONPATH:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: PYTHONPATH
    PYTHON_REPL_SERVER_PORT:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: PYTHON_REPL_SERVER_PORT
    PERPLEXITYAI_API_KEY:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: PERPLEXITYAI_API_KEY
    LANGSMITH_API_KEY:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: LANGSMITH_API_KEY
    LANGSMITH_PROJECT:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: LANGSMITH_PROJECT
    LANGSMITH_DEFAULT_RUN_NAME:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: LANGSMITH_DEFAULT_RUN_NAME
    LANGCHAIN_TRACING_V2:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: LANGCHAIN_TRACING_V2
    LANGCHAIN_ENDPOINT:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: LANGCHAIN_ENDPOINT
    LANGCHAIN_API_KEY:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: LANGCHAIN_API_KEY
    LANGCHAIN_PROJECT:
      valueFrom:
        secretKeyRef:
          name: jupyter-user-secret
          key: LANGCHAIN_PROJECT

cull:
  enabled: true

[szymonorz]

I don't know if you have solved your problem since this is an old issue but I had the exact same issue. It turned out I was overwritting c.JupyterHub.load_roles which removed jupyterhub-idle-culler role. You are doing exactly that with

hub:
  config:
    JupyterHub:
      load_roles:
        - ....

If you want to append to the list of roles you should use

hub:
  loadRoles:
    - .....

which gets evaluated in jupyterhub_config.py here

zero-to-jupyterhub-k8s/jupyterhub/files/hub/jupyterhub_config.py

Line 415 in c895979

for key, role in get_config("hub.loadRoles", {}).items():