feat(glm-5.3-flash): add day-0 serving support by FlamingoPg · Pull Request #1259 · lightseekorg/tokenspeed

FlamingoPg · 2026-08-26T14:17:27Z

Summary

add day-0 GLM-5.3-Flash model, MTP, multimodal, KDA/DSA, and hybrid KV-cache runtime support
add NVIDIA and AMD kernels for KPool, sparse attention, mHC, and GLM-5.3-Flash MoE paths
integrate cancellation, pause/resume, cache transfer, CUDA graph, sampling, and server argument handling
add model/unit tests plus dedicated NVIDIA and AMD CI/evaluation workflows

Test Plan

git diff --check origin/main...shared/glm-5.3-flash
syntax-compile all 117 changed Python files
cold-start ts serve on 4x NVIDIA B200 with TP4/EP4 and FlashInfer 0.6.18rc4
verify L2 cache remains enabled (disable_l2_cache=False)
verify SMG text completion returns HTTP 200
verify a real MMMU-Pro image request returns HTTP 200 with a valid image description

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af1b7fef74

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 628253a894

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

raikonenfnu · 2026-08-27T15:32:10Z

@Max191 ^

Max191 · 2026-08-27T20:16:24Z

@FlamingoPg I rebased this branch to resolve the conflicts. One of the conflicts was with my changes, and the other is on nvidia side. I pushed up the rebase to a separate branch in case you wanted to take a look first: https://github.com/lightseekorg/tokenspeed/tree/Max191/glm-5.3-flash-rebase

I can force push to the shared/glm-5.3-flash branch if it looks good to you. The conflict was pretty simple, but I uploaded the conflict report my agent gave me in case you wanted to take a look at the Deep GEMM changes: https://gist.github.com/Max191/9b84d95713450faf85537dc96dff566c

lightseek-bot · 2026-08-27T20:54:18Z

Hi @Max191 It’s okay to just force-push; LGTM.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59ef15a8c6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5f63be06d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: adc15ebff2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-28T10:29:57Z

+                # target-only topology that plane aliases otherwise-unused KDA
+                # slab space; a merged target+draft plan adds it explicitly.
+                index_plane_id,
+                (pooled_rows, index_head_dim + 4),


Allocate scale bytes for every KPool FP8 group

When a GLM configuration uses a supported index_head_dim above 128, such as 256, dsa_index_k_row_bytes and index_k_block_views require one four-byte scale per 128-value group, but this field reserves only four scale bytes total. The resulting scale view extends beyond the planned index page or aliases other packed storage, causing KPool initialization failures or corrupted index reads/writes; size the field using the full group count.

AGENTS.md reference: AGENTS.md:L23-L35

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a52782bf1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-28T11:13:20Z

+    )
+    kpool = require_positive_int("index_kpool", text_config.index_kpool)
+    index_head_dim = require_positive_int("index_head_dim", text_config.index_head_dim)
+    pooled_rows = GLM53_FLASH_LOGICAL_BLOCK_TOKENS // kpool


Require KPool size to divide the logical block

Reject configurations where index_kpool does not evenly divide 64 before deriving pooled_rows. For example, index_kpool=3 assigns 21 compressed rows to a 64-token cache block, so the pool containing tokens 63–65 is written into the next block; after a 64-token prefix-cache hit, token 63 is neither part of that block nor restored in the request-local tail, causing continuation indexing to use incomplete history. This must preserve the documented separation between logical block boundaries and physical row geometry.

AGENTS.md reference: AGENTS.md:L25-L35

Useful? React with 👍 / 👎.

lightseek-bot · 2026-08-28T22:29:35Z

Hi @FlamingoPg @Max191 please fix the conflicts and ci failures, thanks

chatgpt-codex-connector · 2026-08-29T10:59:02Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-31T16:40:24.590836Z	`de9f442`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 76b6295962

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-29T11:07:37Z

+            raise ValueError(f"layer {layer_id} has no DSA index cache") from exc
+        if buffer is None:
+            raise ValueError(f"layer {layer_id} has no DSA index cache")
+        return buffer


Fence the KPool index before first access

When Host L2 loading restores a GLM-5.3-Flash request, this accessor exposes _index_k without first calling layerwise_load_tracker.wait_for_layer(layer_id), unlike the existing DSA and hybrid-state accessors. KPool reads or writes this side cache before the later MLA-buffer access can establish the fence, so it can race the asynchronous H2D restore and produce stale or corrupted sparse selections; wait on the tracker here before returning the buffer.

AGENTS.md reference: AGENTS.md:L25-L35

Useful? React with 👍 / 👎.

Squash the GLM-5.3-Flash runtime, kernels, tests, and CI integration from the shared development branch. Co-authored-by: Max Dawkins <max.dawkins@gmail.com> Co-authored-by: Stanley Winata <stanley.winata@amd.com> Signed-off-by: FlamingoPg <1106310035@qq.com>

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>

Signed-off-by: FlamingoPg <1106310035@qq.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: de9f442bf8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-31T16:40:21Z

+            pool_size=self.pool_size,
+            page_size=index_cache.shape[1],
+            kv_page_size=ctx.token_to_kv_pool.arena.kv_page_size,
+            topk_pools=self.index_topk // self.pool_size,


Validate KPool top-k divisibility

When a checkpoint sets index_topk to a positive value that is not divisible by index_kpool (for example, 2049 and 4), this truncates the requested count to 512 pools, while Glm53FlashAttention allocates its decode output with index_topk + index_kpool - 1 columns. The expansion kernel instead requires 512 * 4 + 3 = 2051 columns, not the allocated 2052, so the first decode raises an out must be int32 ... error. Reject non-divisible configurations or derive the workspace width from the same pool count.

Useful? React with 👍 / 👎.

FlamingoPg requested review from a team as code owners August 26, 2026 14:17

FlamingoPg mentioned this pull request Aug 26, 2026

docs(glm-5.3-flash): add model recipe #1260

Merged

chatgpt-codex-connector Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread test/ci/eval/glm-5.3-flash-fp8-mtp-tp4ep1-evalscope-aime26-amd.yaml Outdated

FlamingoPg force-pushed the shared/glm-5.3-flash branch from 628253a to af1b7fe Compare August 26, 2026 14:41

chatgpt-codex-connector Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread python/tokenspeed/runtime/engine/request_handler.py Outdated

FlamingoPg mentioned this pull request Aug 26, 2026

docs(glm-5.3-flash): add model recipe #1261

Merged

lightseek-bot added the high priority label Aug 26, 2026

Max191 force-pushed the shared/glm-5.3-flash branch from e322269 to 59ef15a Compare August 27, 2026 20:56

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread python/tokenspeed/runtime/cache/transfer/layout.py Outdated

FlamingoPg force-pushed the shared/glm-5.3-flash branch from b9505e2 to 0c44a48 Compare August 28, 2026 08:42

chatgpt-codex-connector Bot reviewed Aug 28, 2026

View reviewed changes

Comment thread test/ci/README.md Outdated

chatgpt-codex-connector Bot reviewed Aug 28, 2026

View reviewed changes

FlamingoPg force-pushed the shared/glm-5.3-flash branch from adc15eb to 7a52782 Compare August 28, 2026 11:07

chatgpt-codex-connector Bot reviewed Aug 28, 2026

View reviewed changes

FlamingoPg force-pushed the shared/glm-5.3-flash branch from 49283be to 76b6295 Compare August 29, 2026 11:02

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

abatilo mentioned this pull request Aug 30, 2026

Capturable grammar hangs decode CUDA-graph capture when capture batch x draft tokens exceeds 128 logits rows (xgrammar, TP=8, B300) #1314

Open

FlamingoPg and others added 7 commits August 31, 2026 10:29

add import/target guards for amd tests

b3395b2

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>

fix(amd): honor KDA replay group indices

25301d5

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>

fix(glm-5.3-flash): import cache helpers from kernel package

8db7dce

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>

test(amd): gate native FP8 DSA decode to gfx950

572c49b

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>

test(amd): compare KPool FP8 values by quantization steps

9eb77d3

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>

test(kimi): isolate router mock from PDL state

5cfdca7

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>

FlamingoPg added 8 commits August 31, 2026 10:31

fix(glm-5.3-flash): address cross-platform CI failures

b11308c

Signed-off-by: FlamingoPg <1106310035@qq.com>

ci(glm-5.3-flash): resolve checkpoints by model id

93adb77

Signed-off-by: FlamingoPg <1106310035@qq.com>

ci(glm-5.3-flash): use public model repositories

23f6d1d

Signed-off-by: FlamingoPg <1106310035@qq.com>

fix(ci): cache FlashInfer JIT wheel downloads

7c90374

Signed-off-by: FlamingoPg <1106310035@qq.com>

fix(glm-5.3-flash): address review feedback

75daa46

Signed-off-by: FlamingoPg <1106310035@qq.com>

fix(ci): fetch GLM-5.3-Flash tokenizer from Hugging Face

b442ced

Signed-off-by: FlamingoPg <1106310035@qq.com>

fix(ci): extend GLM-5.3-Flash AIME reasoning budget

6385d9f

Signed-off-by: FlamingoPg <1106310035@qq.com>

fix(glm-5.3-flash): simplify integration and CI

de9f442

Signed-off-by: FlamingoPg <1106310035@qq.com>

FlamingoPg force-pushed the shared/glm-5.3-flash branch from 76b6295 to de9f442 Compare August 31, 2026 16:32

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Uh oh!

Conversation

FlamingoPg commented Aug 26, 2026

Summary

Test Plan

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

raikonenfnu commented Aug 27, 2026

Uh oh!

Max191 commented Aug 27, 2026

Uh oh!

lightseek-bot commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

lightseek-bot commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chatgpt-codex-connector Bot commented Aug 29, 2026 •

edited

Loading