Skip to content

fix(scheduler): enforce PodSet gang admission floor - #2096

Open
nvyutwu wants to merge 1 commit into
kai-scheduler:mainfrom
nvyutwu:fix/gang-admission-floor
Open

fix(scheduler): enforce PodSet gang admission floor#2096
nvyutwu wants to merge 1 commit into
kai-scheduler:mainfrom
nvyutwu:fix/gang-admission-floor

Conversation

@nvyutwu

@nvyutwu nvyutwu commented Aug 22, 2026

Copy link
Copy Markdown

Description

GetTasksToAllocate documents that task collection respects minAvailable, but a PodSet below its admission floor currently returns every eligible task in the priority queue up to the remaining requirement. If a caller consumes that under-filled list, checkpoint/rollback is atomic only over the partial list.

This change makes the shared collector return no tasks until enough eligible tasks exist to satisfy the PodSet's remaining minAvailable requirement. Once the PodSet is admitted, existing elastic behavior is unchanged and continues to return at most one additional task.

The allocate action already filters jobs through IsReadyForScheduling. This patch preserves that guard and enforces the same invariant at the lower-level collection boundary used by other scheduler paths.

This is independent of and complementary to #2095, which fixes namespace collisions in PodGroup identity and membership.

Related Issues

N/A. Production investigation: https://reports-a3ad46.gitlab-master-pages.nvidia.com/reports/kai-clique-pinning-prd11/report.html

Checklist

  • Self-reviewed
  • Added/updated tests
  • Updated documentation (not needed; no API or configuration change)
  • Added a changelog fragment via make changelog

Breaking Changes

None.

Additional Notes

Pre-fix red test:

  • minAvailable=2, one pending task: returned 1, expected 0
  • minAvailable=4, three pending tasks: returned 3, expected 0
  • minAvailable=4, two allocated and one pending task: returned 1, expected 0 because two remaining members are required

Post-fix validation on linux/amd64:

  • go test ./pkg/scheduler/api/podgroup_info -count=1
  • go test ./pkg/scheduler/... -timeout 30m -count=1
  • go vet ./...
  • make lint-go (0 issues)
  • make build-go-amd SERVICE_NAME=scheduler

No physical GPU is required for these scheduler-state tests.

@nvyutwu
nvyutwu marked this pull request as ready for review August 23, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant