Commit c4efa33
committed
fix(provisioning,server): exit retry loop on Accepted outcome; await runtime in foreground mode
Two live-AWS regressions, both rooted in earlier work on this branch.
Provisioning retry loop spawned redundant fleets for async providers.
Async providers (EC2Fleet/SpotFleet REQUEST/MAINTAIN) return an
Accepted outcome with zero instances on the first call — fulfilment
finishes out of band and the status poller reconciles it. The retry
loop interpreted "0 fulfilled + not final" as "retry with a new
batch", so each accepted request grew a second and third fleet on the
side. Polling later saw one healthy fleet and N-1 empty fleets and
flipped the request to complete_with_error. Break out of the loop as
soon as an Accepted outcome lands so the single accepted fleet drives
the rest. No in-tree provider emits RequiresFollowUp today; if one
does later, persist_acquiring becomes reachable again.
Foreground server start re-entered asyncio.run inside the CLI's own
asyncio.run(main()). daemon_mod.start called _spawn_runtime which
called asyncio.run, raising RuntimeError "asyncio.run cannot be
called from a running event loop". The exception was caught and
returned exit_code=1, so the subprocess died silently inside 30s and
every rest_api live test hit the "ORB server failed to start" timeout.
handle_server_start now drives the pid + token lifecycle directly in
foreground mode and awaits the runtime coroutine in the existing
event loop.
The retry-loop persist_acquiring tests guarded a write that only ran
between retry attempts. With Accepted no longer triggering a retry,
the persist path is unreachable; the post-call
update_request_from_provisioning in request_creation_handlers already
stamps IN_PROGRESS (or PARTIAL / FAILED) once the orchestrator
returns. Replaced with a test that verifies the Accepted outcome
exits the loop after exactly one provider call.1 parent 4ab16a7 commit c4efa33
5 files changed
Lines changed: 48718 additions & 117 deletions
File tree
- src/orb
- application/services
- interface
- tests/unit
- application/services
- interface
0 commit comments