Skip to content

fix(grouped_gemm): initialize Xe2 scheduler counter before launch - #524

Open
SergiioB wants to merge 1 commit into
vllm-project:mainfrom
SergiioB:fix/xe2-grouped-gemm-atomic-zeros-and-muse-tuple
Open

fix(grouped_gemm): initialize Xe2 scheduler counter before launch#524
SergiioB wants to merge 1 commit into
vllm-project:mainfrom
SergiioB:fix/xe2-grouped-gemm-atomic-zeros-and-muse-tuple

Conversation

@SergiioB

@SergiioB SergiioB commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Purpose

Initialize the Xe2 grouped-GEMM persistent scheduler counter before launching the kernel, and remove the now-redundant in-kernel zero store.

The previous initialization was performed only by work-group 0, local thread 0. There is no device-wide barrier before other work-groups execute atomicAdd, so another work-group can observe stale state, or a late store can reset a counter that has already been incremented.

Using at::zeros queues initialization before the grouped-GEMM launch. The in-kernel atomic_ref::store(0) is removed as requested in review. The local barrier after each work-group obtains a scheduler ticket remains unchanged.

The paged-decode tuple originally included in this PR was cherry-picked and merged separately in #526. This PR is now rebased onto that commit and contains only the grouped-GEMM fix.

Performance consideration

Although the buffer is one int32, at::zeros may add a fixed dispatch/dependency cost, most visible for small grouped-GEMM calls. Removing the in-kernel store avoids paying both initialization paths. This PR does not claim zero overhead without a matched measurement.

Test plan

  • Existing Xe grouped-GEMM unit-test matrix on BMG/PVC CI.
  • Repeated-call correctness and XPU graph replay validation.
  • End-to-end eager and graph-replay latency comparison, emphasizing small-M calls.

Current validation status

  • Branch rebased onto current upstream main / merged feat(attn): add paged-decode tuple 16,128,64,false,true,false #526.
  • paged_decode_default.conf has no PR diff.
  • git diff --check: pass.
  • BMG/PVC builds and unit-test workflows were retriggered by the rewritten branch.
  • Fresh graph/performance measurements are still required before making a quantitative overhead claim.

@jikunshang jikunshang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Liangliang-Ma

Copy link
Copy Markdown
Collaborator

@YizhouZ @baodii PTAL at attn part

// starting tile index (especially under XPU graph replay).
at::Tensor atomic_buffer =
at::empty({static_cast<long>(1)}, ptr_A.options().dtype(at::kInt));
at::zeros({static_cast<long>(1)}, ptr_A.options().dtype(at::kInt));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we change empty to zero here, please remove the zero-initialization operation in the subsequent kernel.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And note: This change is reasonable, but it may result in a slight performance degradation.

@jikunshang

Copy link
Copy Markdown
Member

we may need more evaluation about graph behavior.
I think it's ok to add config in separte PR. I cherry-picked in #526. will add to 0.1.13.x release.

The scheduler counter is allocated with at::empty. The kernel assumes
group 0 stores 0 before other groups atomicAdd, but SYCL does not
guarantee that group 0 runs first. A leftover integer becomes the
starting tile index; XPU graph replay makes this systematic because
the captured buffer address is reused.

at::zeros queues fill_(0) happens-before the GEMM and is part of the
captured graph, so every replay starts at 0. This is scheduler-counter
init only; the GEMM reduction itself is tile-disjoint.

Signed-off-by: SergiioB <sergiobarrientose@outlook.com>
@SergiioB
SergiioB force-pushed the fix/xe2-grouped-gemm-atomic-zeros-and-muse-tuple branch from d34e28a to 4043e76 Compare August 14, 2026 05:26
@SergiioB SergiioB changed the title fix(grouped_gemm): zero Xe2 scheduler counter; add Muse local paged-decode tuple fix(grouped_gemm): initialize Xe2 scheduler counter before launch Aug 14, 2026
@SergiioB

Copy link
Copy Markdown
Contributor Author

Addressed the review comments in the rewritten branch:

  • changed the scheduler buffer to at::zeros;
  • removed the subsequent work-group-0 atomic_ref::store(0) block from MoEGEMM;
  • retained the local barrier after atomicAdd, which is still required to publish the scheduler ticket through SLM;
  • removed the paged-decode change from this PR; it is already merged via feat(attn): add paged-decode tuple 16,128,64,false,true,false #526;
  • rebased onto current upstream main.

The host-side initialization provides launch ordering. The removed in-kernel relaxed store had no device-wide barrier before other work-groups reached atomicAdd, and a late store could reset an already-incremented counter.

I agree that at::zeros may add a small fixed dispatch/dependency cost even though the allocation is one int32. I have not claimed zero overhead in the revised description. The BMG/PVC workflows are running again; I will report explicit graph-replay correctness and matched small-M latency evidence rather than infer the impact.

@SergiioB

SergiioB commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Current status after rechecking the full thread: all recorded BMG/PVC build and unit-test jobs are green, and the paged-decode change was correctly split into merged #526. The remaining review request is still the graph-replay correctness and matched small-M latency evaluation for the host-side at::zeros initialization.

I have not produced that measurement yet, so I am not claiming the PR is ready based on CI alone. Keeping the PR open pending that evidence or maintainer direction on whether the fixed dispatch cost is acceptable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants