Skip to content

test(dashboard-api): cover unhealthy/warnings/retry/catalog regression gaps#1101

Draft
yasinBursali wants to merge 1 commit intoLight-Heart-Labs:mainfrom
yasinBursali:test/pytest-coverage-batch
Draft

test(dashboard-api): cover unhealthy/warnings/retry/catalog regression gaps#1101
yasinBursali wants to merge 1 commit intoLight-Heart-Labs:mainfrom
yasinBursali:test/pytest-coverage-batch

Conversation

@yasinBursali
Copy link
Copy Markdown
Contributor

What

Adds pytest coverage for four dashboard-api / host-agent behaviors that were previously untested:

  • _compute_extension_status's stale-window → "unhealthy" transition for user extensions.
  • enable_extension's multi-service mixed-outcome warnings list (cross-svc accumulator).
  • Host-agent enable-retry edge cases (no-hook, corrupted progress JSON, mid-install setup_hook status).
  • Host-agent _handle_model_download regression shield for the missing/corrupt catalog → 500 contract.

Why

  • The L194 unhealthy return for user extensions lacked a regression test for the stale-started path; only the no-progress-file path was covered.
  • enable_extension accumulates per-svc warnings across multi-service enables, but only single-svc cases were tested. A regression that broke cross-svc warning attribution (warning gets attributed to the wrong svc, or main's failure spuriously suppresses dep warnings) would have shipped silently.
  • The host-agent enable-retry path has three edge cases that aren't pinned by the existing TestEnableRetry class shipping in another in-flight PR: extensions without a post_install hook, progress files that became corrupt mid-install, and extensions retrying while still in a setup_hook mid-install state.
  • The model-download endpoint's missing-catalog and corrupt-catalog responses returned 403 ("Model not in library catalog") rather than 500 ("Model catalog unavailable"), conflating a server-side state issue with a permission denial. The 500 split is shipping in another PR; this lands a regression shield against a future revert.

How

tests/test_extensions.py (+185 lines, 3 new test methods):

  • TestExtensionLifecycleStatus::test_stale_started_unhealthy — fixture uses now-305s for updated_at (must be >300s to fall through the started-recency window AND <3600s to avoid _read_progress suppressing the file as stale-error-only). Asserts the unhealthy path returns "unhealthy" for user extensions whose container is stale-started + reporting unhealthy.
  • TestEnableExtensionHookReturnHandling::test_multi_svc_pre_start_failure_on_dep_does_not_warn — dep's pre_start fails, main's all pass; asserts warnings == [] (terminal pre_start failure isn't accumulated).
  • TestEnableExtensionHookReturnHandling::test_multi_svc_post_start_failure_on_main_only_warns_main — main's post_start fails, dep all pass; asserts exactly one warning naming main, none for dep.

tests/test_host_agent.py (+285 lines, 2 new test classes):

  • TestEnableRetryEdgeCases (named to avoid collision with the in-flight TestEnableRetry class) — three cases for the retry path that exercise no-hook, corrupted-JSON-progress, and mid-install setup_hook states.
  • TestModelDownloadCatalogUnavailable — three cases asserting 500 for missing catalog, 500 for corrupt catalog, and 403 preserved for clean catalog with model not listed.

All tests use existing helper patterns (_setup_user_ext, _patch_mutation_config, in-process HTTPServer(("127.0.0.1", 0), _mod.AgentHandler) wire pattern, monkeypatch for module attribute changes) and run in tmp_path to avoid global state pollution.

Testing

On upstream/main (current state):

  • 4 new tests pass: stale_started_unhealthy + 2× multi-svc warnings + TestModelDownloadCatalogUnavailable case (c) clean-catalog-not-listed.
  • 5 new tests fail-as-expected (DRAFT-only):
    • AttributeError on _read_progress_status (TestEnableRetryEdgeCases — the helper is added by another in-flight PR).
    • 403 != 500 (TestModelDownloadCatalogUnavailable cases a and b — the 500 split lands in another in-flight PR).

After the prerequisite branches merge, the rebase will turn all 9 tests green.

Pre-existing 12 baseline failures in the full pytest suite (Python 3.14 / asyncio compat) verified unchanged via stash-pop on the PR-Y worktree — none introduced by this PR.

Review

Critique Guardian: APPROVED. Falsifiability verified across all 9 cases (assertions on response shape, content, and warnings-list contents — not just status codes).

Known Considerations

  • This PR is DRAFT and must merge after the in-flight host-agent enable-retry branch and the host-agent runtime-hygiene branch land. The 5 fail-as-expected tests are the mechanical safeguard.
  • Class-name collision avoided: TestEnableRetryEdgeCases (this PR) vs TestEnableRetry (other branch).
  • A small coverage gap not pursued here: simultaneous dep pre_start + main post_start failures combined into one call. Edge-of-edge case; could be a future addition.

Platform Impact

  • macOS: no impact — pytest runs identically; mocks shield against GPU_BACKEND-driven branching.
  • Linux: no impact — same.
  • Windows-WSL2: no impact — same.

…n gaps

Pytest additions across test_extensions.py and test_host_agent.py:

- TestExtensionLifecycleStatus::test_stale_started_unhealthy — locks
  in the stale-window -> "unhealthy" transition for user extensions in
  _compute_extension_status. Uses now-305s for the progress fixture so
  the file falls through the started-recency window without tripping
  _read_progress's >3600s suppression.
- TestEnableExtensionHookReturnHandling: two new cases for multi-svc
  mixed-outcome warnings — dep pre_start failure does not accumulate;
  main post_start failure is attributed to main only.
- TestEnableRetryEdgeCases: three cases covering no-hook retry, corrupt
  progress JSON falling back to sync path, and mid-install setup_hook
  status routing through sync. Class name avoids collision with the
  TestEnableRetry class shipping in another in-flight branch.
- TestModelDownloadCatalogUnavailable: three cases asserting 500 for
  missing catalog, 500 for corrupt catalog, and 403 preserved for
  clean catalog with model not listed.

5 of 9 new tests fail-as-expected on current upstream/main; they pass
once the prerequisite branches (host-agent enable-retry; host-agent
runtime-hygiene) land. PR is filed DRAFT for that reason.

No production code touched. 12 pre-existing baseline failures in the
full suite (Python 3.14 asyncio compat) are unchanged.
@Lightheartdevs
Copy link
Copy Markdown
Collaborator

Merge-pass recheck: holding this as draft/red.

The regression coverage is valuable, but the PR body says 5 tests are fail-as-expected until prerequisite branches land, and the live api check is currently failing. Please rebase after the prerequisite host-agent/runtime work is merged, make the full API job pass against current main, and then undraft for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants