fix(BA-7328): stop scheduling past a resource-exhausted session

[HyeockJinKim]

Summary

• The workload loop in SessionProvisioner.schedule_resource_group skipped every failure, so lower-priority sessions kept taking the resources a blocked higher-priority session was waiting for. The pass now stops on resource exhaustion only — an architecture mismatch, an unsatisfied dependency, an exceeded quota or a group with no agents is specific to the session that hit it and must not stall the queue.
• Sessions left unattempted are reported as skips instead of failures, so they no longer accumulate retry pressure for work they never got to do. session_pending_queue now returns the whole waiting queue in sequencing order rather than only the failed sessions.
• Skips are recorded and counted on their own scheduling-history record, and the give-up classification ignores them, so a session queued behind a blocked one is never deprioritized. The last phase record is carried onto the session as one LastPhase value instead of three loose fields.

Test plan

• A big high-priority session that does not fit blocks the small low-priority session behind it, which is reported as a skip
• The per-agent container cap blocks the queue the same way an exhausted slot does
• An architecture mismatch, an unsatisfied dependency, an exceeded quota, a reserved batch session not yet due, and a resource group with no agents each leave the sessions behind schedulable
• session_pending_queue returns the blocked session and every skipped session in sequencing order
• Repeated skips accumulate on their own history record while the attempt record keeps the count give-up may see
• However many skips a session accumulates, it is classified need_retry rather than give_up
• Attempt results still merge into one history record, so give-up keeps firing for handlers that declare no transition for it

Resolves BA-7328

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Updates Sokovan scheduling to stop after resource exhaustion while allowing session-specific failures to continue through the queue.

Changes:

• Reports unattempted sessions as skips and preserves pending-queue order.
• Separates skip history from scheduling attempts.
• Consolidates retry history into LastPhase with expanded tests.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
changes/13707.fix.md	Adds the fix changelog.
src/ai/backend/manager/models/scheduling_history/row.py	Separates skip and attempt history records.
src/ai/backend/manager/sokovan/scheduler/coordinator.py	Uses LastPhase for failure classification.
src/ai/backend/manager/sokovan/scheduler/handlers/lifecycle/schedule_sessions.py	Maps provisioner skips to skipped transitions.
src/ai/backend/manager/sokovan/scheduler/provisioner/provisioner.py	Stops scheduling after resource exhaustion.
src/ai/backend/manager/sokovan/scheduler/results.py	Adds scheduling-skip results.
src/ai/backend/manager/views/sokovan/lifecycle.py	Introduces LastPhase.
tests/unit/manager/models/test_scheduling_history_row.py	Tests history merge rules.
tests/unit/manager/repositories/scheduler/test_update_with_history.py	Tests separate skip history.
tests/unit/manager/sokovan/scheduler/handlers/conftest.py	Updates session fixtures.
tests/unit/manager/sokovan/scheduler/handlers/test_lifecycle_handlers.py	Tests skip propagation.
tests/unit/manager/sokovan/scheduler/provisioner/conftest.py	Expands scheduling fixtures.
tests/unit/manager/sokovan/scheduler/provisioner/test_provisioner.py	Tests blocking and non-blocking failures.
tests/unit/manager/sokovan/scheduler/test_coordinator.py	Tests phase history and classification.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[fregataa]

Could you check this review?

[HyeockJinKim]

Wrong response. Since I get it wrong every time I say it, I kind of just want to leave it out.

[fregataa]

get_last_session_histories() returns only the latest history record of a session and it seems possible to get only a attempt record, ignoring skip records

[fregataa]

How about adding a test to check a session with "attempt -> skip -> attempt"?

[github-actions]

The backport to 26.4 hit conflicts: #13717 was created with the conflict markers committed in:

src/ai/backend/manager/sokovan/data/lifecycle.py
src/ai/backend/manager/sokovan/scheduler/coordinator.py
src/ai/backend/manager/sokovan/scheduler/provisioner/provisioner.py
src/ai/backend/manager/sokovan/scheduler/results.py
tests/unit/manager/sokovan/scheduler/handlers/test_lifecycle_handlers.py
tests/unit/manager/sokovan/scheduler/provisioner/conftest.py
tests/unit/manager/sokovan/scheduler/provisioner/test_provisioner.py

CI on it fails until the markers are resolved; check out its branch, fix, push, and merge it by hand (auto-merge is not enabled for conflicted backports).