What is the problem you're trying to solve

Volcano has several related efforts to improve preemption and reclaim observability, but they answer different operational questions.

Relationship to existing work

This proposal overlaps with several ongoing observability efforts, but the metrics have different counting units and answer different operational questions.

The following comparison describes those changes as currently proposed:

These signals are intentionally not interchangeable:

• An attempt can occur without a committed eviction.
• One committed transaction can contain multiple victim Pods.
• A per-victim attribution metric can emit multiple labeled series for one transaction.
• A victim Gauge describes the latest observation rather than a cumulative number of committed decisions.
• A committed scheduler decision does not guarantee that every asynchronous Kubernetes Pod deletion eventually succeeded.

These are useful, but Volcano still lacks one low-cardinality, monotonic metric that answers a narrower question consistently across all four actions:

How many scheduler decisions containing eviction operations reached the commit boundary?

The existing volcano_total_preemption_attempts counter cannot answer this. An attempt can evaluate many candidates without committing any eviction. In a local reproduction, this counter had reached 100 before the first real preemption occurred.

The victim Gauges answer a different question: how many victim Pods were selected for the latest observation. They do not count how many committed eviction decisions occurred, and a Gauge can retain or overwrite a previous value.

Action latency metrics show how long an action ran, but not whether it committed an eviction.

It is useful to treat eviction observability as separate layers:

Describe the solution you'd like

Add a Counter:

volcano_eviction_transactions_total{action}

The bounded action label has four supported values:

The metric has the following precise semantics:

• Increment once when a scheduler Statement containing at least one eviction operation reaches Statement.Commit().
• Do not increment for a discarded Statement.
• Do not increment for a committed Statement with no eviction operation.
• Increment once when one committed Statement contains multiple victim Pods.
• Count the scheduler's committed decision, not individual Pods and not the number of node or candidate evaluations.
• Do not claim that every asynchronous Kubernetes Pod deletion subsequently completed successfully.

This commit boundary gives the Counter an objective and reproducible meaning. It avoids inflation from scheduler retries and node scans, avoids stale last-observation Gauge semantics, and allows the same PromQL aggregation across all four actions.

Examples:

# All committed eviction transactions
sum(rate(volcano_eviction_transactions_total[5m]))

# Non-gang cross-queue reclaim
rate(volcano_eviction_transactions_total{action="reclaim"}[5m])

# All gang-level eviction transactions
sum(rate(volcano_eviction_transactions_total{
  action=~"gangpreempt|gangreclaim"
}[5m]))

No separate scope or gang labels are proposed because those dimensions are already deterministically represented by the low-cardinality action label.

Queue names are intentionally not proposed in this first version. One transaction may contain victims from multiple queues, so a source_queue/victim_queue Counter needs separate semantics to avoid double-counting and unnecessary label cardinality.

Goals and non-goals

Goals:

• Provide one comparable committed-decision Counter for preempt, reclaim, gangpreempt, and gangreclaim.
• Distinguish same-queue/cross-queue and regular/gang behavior through the action label.
• Keep label cardinality bounded.
• Update the metric only at the Statement commit boundary.
• Document and test the exact counting unit.

Non-goals:

• Replacing the attempt or victim metrics proposed by the related work.
• Counting victim Pods or reclaimed resource quantities.
• Adding Pod, Job, Node, or Queue labels.
• Reporting final Kubernetes API deletion success.
• Instrumenting the shuffle action in this change.

How the metrics complement each other

The related metrics cover different layers of eviction observability:

1. Attempt metrics describe scheduler effort.
2. volcano_eviction_transactions_total describes committed scheduler decisions containing evictions.
3. Victim metrics describe the size of the selected impact.
4. Pod, Node, and Queue labels provide attribution.
5. Final Kubernetes deletion outcomes would require separate instrumentation and are not covered by these metrics.

For example, an operator investigating cross-queue reclaim could correlate:

volcano_total_reclaim_attempts

with:

rate(volcano_eviction_transactions_total{action="reclaim"}[5m])

and, if the related victim metrics are available, the reclaim victim-count distribution.

The metrics should not be directly divided into a success ratio unless their attempt and transaction units are first confirmed to match. A task-level attempt and a committed Statement are not necessarily one-to-one.

This proposal does not supersede or close #4040, #5223, or #5661. A future pull request implementing this proposal should link only this issue with Fixes #5753 and reference the other work as complementary.

Acceptance criteria

• A discarded Statement does not increment the Counter.
• A committed Statement without evictions does not increment the Counter.
• A committed Statement with one or more evictions increments it exactly once.
• All four supported scheduler actions use the same metric.
• Unit tests cover the metric helper and eviction detection on a Statement.
• The metric is documented in docs/design/metrics.md.
• The implementation is validated with workloads that trigger all four action values on a Kubernetes cluster.

Additional context

A prototype was implemented from Volcano commit 2a5d61cbc1b3ead22c50d49afe9cb98d144d0bf0.

Local checks:

hack/verify-gofmt.sh
git diff --check
go test -count=1 \
  ./pkg/scheduler/metrics \
  ./pkg/scheduler/framework \
  ./pkg/scheduler/actions/preempt \
  ./pkg/scheduler/actions/reclaim \
  ./pkg/scheduler/actions/gangpreempt \
  ./pkg/scheduler/actions/gangreclaim

The locally built scheduler was then deployed to a one-node kind Kubernetes v1.36.1 cluster. Independent workloads triggered each path:

The counters remained unchanged during subsequent idle scheduling cycles.

AI assistance disclosure: OpenAI Codex was used to inspect the scheduler code, prepare and test the prototype, construct the kind validation scenarios, and help draft this issue. I reviewed the implementation and verified the claims using source inspection, unit tests, and an actual Kubernetes cluster.

Documentation Updates

• This feature requires design or user documentation changes.
• If documentation changes are required, I will ensure the relevant documents are updated and published to the Volcano official website via the volcano-sh/website repository.