[BUG]: GET /Jobs endpoint returns 404 errors

[khaledosman]

Description

Opening lumigator throw 500 server errors when accessing the GET /jobs endpoint.

I tried deleting all datasets and experiments with no luck, also tried make clean-all, but it seems like the jobs are partially persisted somewhere else somehow leading to this error

Reproduction

1. Run make clean-all
2. Run make local-up
3. Open lumigator frontend localhost:80 or localhost:3000 if running frontend dev server via npm run dev

Relevant log output

backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
backend-1      |     result = await app(  # type: ignore[func-returns-value]
backend-1      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
backend-1      |     return await self.app(scope, receive, send)
backend-1      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
backend-1      |     await super().__call__(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
backend-1      |     await self.middleware_stack(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
backend-1      |     raise exc
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
backend-1      |     await self.app(scope, receive, _send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
backend-1      |     await self.simple_response(scope, receive, send, request_headers=headers)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 144, in simple_response
backend-1      |     await self.app(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
backend-1      |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend-1      |     raise exc
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
backend-1      |     await app(scope, receive, sender)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
backend-1      |     await self.middleware_stack(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
backend-1      |     await route.handle(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
backend-1      |     await self.app(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
backend-1      |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend-1      |     raise exc
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
backend-1      |     await app(scope, receive, sender)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
backend-1      |     response = await f(request)
backend-1      |                ^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
backend-1      |     raw_response = await run_endpoint_function(
backend-1      |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
backend-1      |     return await run_in_threadpool(dependant.call, **values)
backend-1      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
backend-1      |     return await anyio.to_thread.run_sync(func)
backend-1      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
backend-1      |     return await get_async_backend().run_sync_in_worker_thread(
backend-1      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
backend-1      |     return await future
backend-1      |            ^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 962, in run
backend-1      |     result = context.run(func, *args)
backend-1      |              ^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/backend/api/routes/jobs.py", line 125, in list_jobs
backend-1      |     jobs = service.list_jobs(skip, limit, job_types)
backend-1      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/backend/services/jobs.py", line 569, in list_jobs
backend-1      |     items=[self.get_job(record.id) for record in records],
backend-1      |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/backend/services/jobs.py", line 569, in <listcomp>
backend-1      |     items=[self.get_job(record.id) for record in records],
backend-1      |            ^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/backend/services/jobs.py", line 543, in get_job
backend-1      |     job_status = self.ray_client.get_job_status(job_id)
backend-1      |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/ray/dashboard/modules/job/sdk.py", line 424, in get_job_status
backend-1      |     return self.get_job_info(job_id).status
backend-1      |            ^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/ray/dashboard/modules/job/sdk.py", line 359, in get_job_info
backend-1      |     self._raise_error(r)
backend-1      |   File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 283, in _raise_error
backend-1      |     raise RuntimeError(
backend-1      | RuntimeError: Request failed with status code 404: Job 67036c9b-f261-43a8-8831-152ec514f67b does not exist.
backend-1      | INFO:     192.168.65.1:36931 - "GET /api/v1/datasets/ HTTP/1.1" 200 OK

Expected behavior

Jobs endpoint should not fail

System Info

• main branch
• macOS

Have you searched for similar issues before submitting this one?

• Yes, I have searched for similar issues