fix(serve): persist running->ready promotion in CLI liveness refresh#101
Open
michaelroy-amd wants to merge 3 commits into
Open
fix(serve): persist running->ready promotion in CLI liveness refresh#101michaelroy-amd wants to merge 3 commits into
michaelroy-amd wants to merge 3 commits into
Conversation
refresh_managed_service_runtime_liveness() ran the real HTTP model-ready probe but only used a passing result to skip demotion, never to persist the running->ready transition. Its twin in providers.rs::ready_local_services() already promotes correctly; mirror that here so load_managed_services() (and therefore the "services" MCP tool and chat's pick_managed_chat_endpoint, which requires exact status "ready") see the true state instead of a manifest stuck at "running" forever. Relates to EAI-7352 Signed-off-by: Michael Roy <michael.roy@amd.com>
3 tasks
load_managed_services_promotes_running_to_ready_once_probe_passes re-read the record via load_managed_service() to check persistence, but that function itself calls refresh_managed_service_runtime_liveness() on every read -- including on an already-"ready" record, since the endpoint-ready check also matches "ready". So the second assertion would pass even if load_managed_services() never wrote the promotion to disk; it wasn't actually proving persistence. Add a direct read of the manifest file's raw JSON in between the two calls and assert its status field is exactly "ready", which only load_managed_services() could have produced. Relates to EAI-7352 Signed-off-by: Michael Roy <michael.roy@amd.com>
…o dep change) Signed-off-by: Michael Roy <michael.roy@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
refresh_managed_service_runtime_liveness()inapps/rocm/src/main.rsalready runs the real HTTP readiness probe (managed_service_endpoint_model_ready()), but only used a passing probe to skip demotion — it never persisted therunning->readytransition. This means a service manifest can sit forever atstatus: "running"even after the model has actually finished loading and is serving requests.Root cause
The twin code path in
apps/rocm/src/providers.rs::ready_local_services()already does this correctly: on a passing probe it setsrecord.status = "ready"and writes the manifest back to disk.refresh_managed_service_runtime_liveness()(used byload_managed_services(), which backs theservicesMCP tool and — viapick_managed_chat_endpoint()— chat's endpoint selection, which requires the exact string"ready") never got the same fix. As a result chat could never discover a genuinely ready local server if the record had been left at"running".Fix
Mirror
ready_local_services()'s promotion: when the probe passes andrecord.status == "running", set it to"ready"and signal a change so the existing write-back logic inload_managed_services()/load_managed_service()persists it.Test Plan
load_managed_services_promotes_running_to_ready_once_probe_passes— spins up a mock/v1/modelsHTTP endpoint, writes a manifest withstatus: "running", callsload_managed_services, and asserts the in-memory and on-disk (load_managed_servicere-read) status is"ready".cargo build -p rocmcargo clippy -p rocm --all-targets --all-features -- -D warningscargo test -p rocm -- --test-threads=1(338 passed)Relates to EAI-7352