Skip to content

Starting user server fails with 500 Internal Server Error after removing profileList #3297

Open
@nocnokneo

Description

@nocnokneo

Bug description

I started with a working multi-profile installation with a few profiles like:

      profileList:
      - display_name: "Standard Environment"
        default: true
      - display_name: "Power User Environment"
        kubespawner_override:
          mem_guarantee: "20G"
          mem_limit: "20G"

I then deleted the entire profileList (no longer want users to have the option of different environments). Now when I try to login and start a singleuser server I get "500 : Internal Server Error". I tested with another user account who had never previously logged in to the multi-profile deployment that worked.

From the logs (see below) it's clear that JupyterHub is remembering the standard-environment slug corresponding to the the first profile I had previously in the multi-profile configuration.

How to reproduce

See explanation above.

Expected behaviour

Expected that users who had logged in previously would still be able to start their singleuser server.

Your personal set up

Z2JH in AWS. Kubernetes version 1.28.

kubectl logs hub-...
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2023-12-08 18:18:31.931 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-12-08 18:18:31.931 JupyterHub app:2889] Using Authenticator: cytohub.authenticator.CytoTronicsOAuthenticator
[I 2023-12-08 18:18:31.932 JupyterHub app:2889] Using Spawner: kubespawner.spawner.KubeSpawner-6.2.0
[I 2023-12-08 18:18:31.932 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2023-12-08 18:18:32.172 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-12-08 18:18:32.241 JupyterHub app:2928] Initialized 0 spawners in 0.003 seconds
[I 2023-12-08 18:18:32.248 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-12-08 18:18:32.250 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2023-12-08 18:18:32.251 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.thirty_days
[I 2023-12-08 18:18:32.251 JupyterHub app:3142] Not starting proxy
[I 2023-12-08 18:18:32.259 JupyterHub app:3178] Hub API listening on http://:8081/hub/
[I 2023-12-08 18:18:32.259 JupyterHub app:3180] Private Hub API connect url http://hub:8081/hub/
[I 2023-12-08 18:18:32.259 JupyterHub app:3189] Starting managed service jupyterhub-idle-culler
[I 2023-12-08 18:18:32.259 JupyterHub service:385] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=18000', '--cull-every=300', '--concurrency=10']
[I 2023-12-08 18:18:32.261 JupyterHub service:133] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=18000 --cull-every=300 --concurrency=10
[I 2023-12-08 18:18:32.264 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[I 2023-12-08 18:18:32.640 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 10.41ms
[I 2023-12-08 18:18:32.652 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 9.11ms
[I 2023-12-08 18:18:42.379 JupyterHub log:191] 200 GET /hub/home (taylor [email protected]) 65.63ms
[I 2023-12-08 18:18:43.642 JupyterHub provider:659] Creating oauth client jupyterhub-user-taylor%20braun-jones
[E 2023-12-08 18:18:43.668 JupyterHub user:884] Unhandled error starting taylor braun-jones's server: No such profile: standard-environment. Options include:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2667, in _start
        await self.load_user_options()
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 3327, in load_user_options
        self._validate_user_options(profile_list)
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 3132, in _validate_user_options
        profile = self._get_profile(profile_slug, profile_list)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 3176, in _get_profile
        raise ValueError(
    ValueError: No such profile: standard-environment. Options include: 
    
[I 2023-12-08 18:18:43.722 JupyterHub reflector:282] watching for pods with label selector='component=singleuser-server' in namespace computeapi
[E 2023-12-08 18:18:43.724 JupyterHub pages:313] Error starting server taylor braun-jones: No such profile: standard-environment. Options include:
    Traceback (most recent call last):
    NoneType: None
    
[W 2023-12-08 18:18:43.724 JupyterHub web:1869] 500 GET /hub/spawn/taylor%20braun-jones (10.1.0.64): Unhandled error starting server taylor braun-jones
[E 2023-12-08 18:18:43.754 JupyterHub log:183] {
      "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
      "Accept-Language": "en-US,en;q=0.9,fr-CH;q=0.8,fr;q=0.7,es-MX;q=0.6,es;q=0.5",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://voilahub.test.cytotronics.io/hub/home",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "same-origin",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
      "Dnt": "1",
      "Upgrade-Insecure-Requests": "1",
      "Sec-Ch-Ua-Platform": "\"Linux\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua": "\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\"",
      "X-Scheme": "https",
      "X-Forwarded-Scheme": "https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "voilahub.test.cytotronics.io",
      "X-Forwarded-For": "10.1.0.64,::ffff:10.1.20.18",
      "X-Real-Ip": "10.1.0.64",
      "X-Request-Id": "178455466202853493bca02ad07e86d7",
      "Host": "voilahub.test.cytotronics.io",
      "Connection": "keep-alive"
    }
[E 2023-12-08 18:18:43.754 JupyterHub log:191] 500 GET /hub/spawn/taylor%20braun-jones (taylor [email protected]) 141.01ms

...

[I 2023-12-08 18:58:21.798 JupyterHub log:191] 200 POST /hub/api/users/labuser/activity ([email protected]) 11.40ms
[I 2023-12-08 18:58:32.690 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 8.89ms
[I 2023-12-08 18:58:32.695 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 4.62ms
[I 2023-12-08 19:03:23.666 JupyterHub log:191] 200 POST /hub/api/users/labuser/activity ([email protected]) 11.24ms
[I 2023-12-08 19:03:32.667 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 7.86ms
[I 2023-12-08 19:03:32.673 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 4.42ms
[I 2023-12-08 19:08:32.724 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 8.23ms
[I 2023-12-08 19:08:32.737 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 11.57ms
[I 2023-12-08 19:08:45.733 JupyterHub log:191] 200 POST /hub/api/users/labuser/activity ([email protected]) 9.49ms

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions