Per-cycle backend maintenance only runs on scale-up cycles

[dotsdl]

Summary

Backend housekeeping in ScriptTemplateHPCManager.create_compute_services (health checks, server reconciliation, clearing completed jobs) only runs on scale-up cycles. On cycles where the queue is drained, the manager is at capacity, or sizing collapses to target == 0, the maintenance never runs. Surfaced while reviewing #3 (adopting the upstream ComputeManager sizing logic).

Details

After #3, the base ComputeManager.cycle() (alchemiscale compute/manager.py) is the only caller of create_compute_services, and it gates the call:

if total_services < self.settings.max_compute_services and num_tasks > 0:
    target = self._compute_jobs_to_create(...)
    if target > 0:
        self.create_compute_services(data, target)   # sole call site

But ScriptTemplateHPCManager.create_compute_services (alchemiscale_hpc/base.py) runs per-cycle housekeeping before the submit loop:

self.batch_api.check_job_health()
self.batch_api.verify_running_jobs(server_job_names)
self.batch_api.clear_successful_jobs()
if self.batch_api.jobs_pending():
    return 0
# ... submit up to `target` scripts

There is no other maintenance hook in cycle(). So whenever num_tasks == 0, the manager is at capacity, or target == 0:

• check_job_health() does not run
• verify_running_jobs() does not reconcile running jobs against server state
• clear_successful_jobs() does not clear completed jobs

This matters most while the task queue drains: completed jobs aren't cleared and running jobs aren't reconciled until new tasks arrive and scale-up triggers again.

The health checks have always lived inside create_compute_services, but adopting the #502 base — which added the num_tasks > 0 and target > 0 gating — is what couples maintenance to the scale-up decision. Worth confirming against pre-#502 behavior: if create_compute_services previously ran every OK cycle, this is a behavioral regression at idle.

Test gap

The unit tests test_create_compute_services (for verify_running_jobs / clear_successful_jobs) call create_compute_services(..., num_tasks=0, target=0) and assert the maintenance fires. The base will never call the method in that state, so these green tests assert behavior production never reaches — masking the gap above.

Suggested fix

Split the housekeeping out of create_compute_services into a dedicated maintenance step that the base ComputeManager.cycle() invokes every OK cycle (regardless of target), leaving create_compute_services purely about submitting target scripts. This likely requires a coordinated change to the upstream alchemiscale.compute.manager.ComputeManager base, so it may be cleaner as a follow-up than to fold into #3.

Re-home the two affected unit tests onto the new maintenance hook so they exercise a reachable code path.

Acceptance criteria

• Confirm whether idle-cycle maintenance was the pre-#502 behavior (regression vs. latent)
• Backend health checks / verify_running_jobs / clear_successful_jobs run every OK cycle, independent of scale-up
• create_compute_services is responsible only for submitting target scripts
• Maintenance unit tests target a code path the production cycle actually exercises

Found during review of #3.