[oscarlius]

What this changes

With MAYA_GLM_ROW_HALVES=1, multi-token GGML CPU prefill still dispatches row blocks through the ordinary queue although the weight rows are split across NUMA nodes. This adds an off-by-default MAYA_GLM_PREFILL_ROW_LOCAL=1 scheduler that prefers the worker's half, then steals from the other half, after validating the live RAM allocation, layer layout and block boundaries.

The patch preserves flat job IDs, kernels, arithmetic and the two down-batch/upload boundaries. Single-token and EXL3 paths are untouched. Failed eligibility checks retain the ordinary queue. Head f3c48a7e25d1206c9d6d2deb3c4aa62728ed42e8 contains this prefill change, regression coverage, CI/CMake wiring and documentation; local 144-thread CPU-pool overlays and dynamic DFlash fixes are excluded.

Draft for technical review. Full clean-branch CUDA build and an isolated four-arm run are complete, but the predeclared whole-model quality gate failed. Not accepted for merge or production promotion. This update replaces the earlier “clean CUDA validation pending” status. The setting remains off by default.

Answers

• The default answers are unchanged (same tokens as before).
• It can change answers, and it is opt-in (off by default). Setting: MAYA_GLM_PREFILL_ROW_LOCAL=1, with MAYA_GLM_ROW_HALVES=1.

The disabled path delegates to the original queue; synthetic CPU fixtures pass bitwise comparisons. Changed timing can alter adaptive CPU/GPU expert dispatch. Whole-model identical tokens and quality preservation are not established. Baseline repeated answers also varied; the quality failure alone does not identify this scheduler as its cause.

Tested on

Latest clean-branch run, 2026-11-13:

• Exact head above; tree 290b19d535f550556904dfa7a390c8913bde10d0; binary SHA256 9ac6c16ac282cc7aea309561444a480077112ee457203f556bedde4e4eb7161b.
• Pinned llama dependency 3cf03257f219afbe7334045ff7c6a06ac68c627d; CUDA 13.0.88, GCC 13.3.0, CMake 3.31.10; Release, sm_89. Full Maya and host/native regression targets built and the regression test passed. Source/dependency files were checked against frozen manifests. No local CPU-pool or DFlash overlays.
• Linux, dual Xeon Platinum 8380, 4×RTX 4090 24 GB at 255 W each, GLM-5.3-Flash, 160 GB RAM tier; IQ3_S gate/up with Q5_K or IQ4_XS down. Clean upstream uses a shared 64-thread pool; four service slots disable DFlash/MTP, verified in startup logs and zero draft counts. Fixed layer boundaries 13,24,35.
• Same binary, inputs, frozen profile and configuration for A1 → B1 → B2 → A2. Row-halves stayed enabled; only row-local varied 0/1/1/0. Each arm was a fresh process.
• Each arm: eight sequential plus eight four-client performance requests with 256 output tokens; then the same 16 cases for natural-stop quality with a prospectively frozen 2,048-token cap. All 128 requests retained; no extra arm or replacement samples.

Existing coverage also includes native Windows MSVC/CTest host scheduling, synthetic native CPU parity for Q8_0, Q5_K, packed Q2_K/AVX-512 and both production weight-format pairs, sentinel/write-range checks, HIP-name compatibility and whitespace checks. The CPU fixtures do not prove full worker-chain, CPU/GPU or physical NUMA-placement equivalence. HIP and other NVIDIA architectures remain untested; the HIP-name check is not a HIP build.

Speed (isolated clean-branch observations)

Throughput below is output tokens divided by segment request-drain wall time, including prompt processing, not native decode speed. Sequential prefill uses newly processed input tokens divided by native prompt time. Each table cell covers eight requests.

Arm	Sequential prefill tok/s	Sequential output tok/s	Four-client output tok/s
A1, off	516.77	34.32	110.44
B1, on	574.04	37.67	117.09
B2, on	540.09	36.38	119.22
A2, off	551.34	37.95	113.45

Pooling raw tokens and times across both runs per side:

• Sequential prefill: 533.50 → 556.55 tok/s, +4.32%. A2/A1 drift was +6.69%; B2 was 2.04% slower than A2. Stable sequential gain is not established.
• Sequential output: 36.04 → 37.01 tok/s, +2.70%; A2/A1 drift was +10.58%.
• Four-client output: 111.93 → 118.15 tok/s, +5.56%. Both B runs exceeded both A runs; all four B/A comparisons ranged from +3.21% to +7.95%. Two runs per side do not establish statistical significance or general benefit.

Sequential requests processed 6,027 new input tokens per arm, with no cache reuse. The four-client segment processed only 461 new tokens and reused 5,566 tokens. Its positive result applies to this short, cache-heavy workload, not long-context cold-prefill throughput.

A separate 144-thread, dynamic-DFlash deployment-overlay experiment isolated the same switch and observed sequential prefill +5.60%, sequential output +3.13%, and four-client output +3.37%. Its relative text scores passed, but 6/64 quality answers hit the 1,024-token cap, so full natural-stop acceptance was not established. Those numbers must not be pooled with the clean 64-thread/no-draft experiment. Older combined-policy +12–25% observations are not the isolated benefit of this PR; the repository document retains that historical record.

Original 64-answer quality study: not accepted

All 64/64 original clean-branch quality answers stopped naturally with no 2,048-token truncation. Raw answers and original locked scoring artifacts are retained.

Two independently confirmed counterexamples still fail the original relative task gate:

• Sequential cache review: both off answers scored 2; one on answer placed a two-party barrier inside a lock-held fetch and then claimed the patched test would pass.
• Four-client metrics document: both off answers scored 2; one on answer retained a claim that latency excluded queue/prompt time alongside the correct send-to-last-token definition.

Scoring-consistency correction: the follow-up found that three original on lease-rules answers had been marked too strictly for equivalent numeric injected-clock tests. A supplementary review scores those three 2; the fourth remains 1 because it omits the owner-first joint boundary. The old “all four are incomplete” wording is corrected. The complete 64-answer set has not been rescored, so the original aggregate mean/pass-rate table is not presented here as a freshly adjudicated comparison. The original whole-model gate still fails on the independently confirmed counterexamples above, and the remaining incomplete lease answer still fails its all-on-complete rule.

Bounded independent quality follow-up, 2026-11-14: gate failed

One new A1 → B1 → B2 → A2 sequence used the same clean binary, 64 shared threads, four slots, fixed 13,24,35 layout and frozen inputs/profile; only row-local varied. Each fresh process answered two serial target cases (cache-review, lease-rules) and the original two four-client companion waves: four loads, 40 answers total, all natural stop, zero 2,048-token cap hits. No performance segment, replacement answer or extra arm was run. This preserves the targeted prompts and concurrency companions, but omits the former performance/complete-serial warmup history; it is not an exact replay of the original internal runtime state or a replacement for the complete 64-answer study.

24 code and 16 document answers were anonymized. Semantic-equivalence inconsistencies in six code scores were corrected and documented before score lock and unblinding, under the unchanged rubric. Explicit numeric injected-time tests and explicit owner → expiry → TTL precedence were accepted as allowed by that rubric; the original and final scoring records are retained.

Mode	Answers per side	Off mean / 2	On mean / 2	Off full-pass rate	On full-pass rate
Serial target cases	4	1.500	1.000	75.00%	50.00%
Four clients	16	1.875	1.875	93.75%	93.75%

• Serial lease-rules fails: off [2,2], on [0,2]. The failing on answer asserts that renewal must never shorten expiry: for old expiry 1000, now=500, ttl=100, it requires expiry ≥1000, whereas the supplied function correctly sets it to 600. Only 3/4 on lease answers are complete, so the special all-four-complete rule also fails.
• Cache-review scores are off [2,0], on [0,2] in both modes. Invalid lock/barrier or lock/event tests occur on both sides. Their errors are retained; these relative pairs tie and pass this follow-up's pair gate.
• The old metrics-document contradiction did not recur in this sample: all four metrics answers and all 16 document answers scored 2. Overall, 9/10 task/mode pairs pass, but the frozen target gate fails.

This is non-acceptance under the declared quality gate, not proof of scheduler-induced quality regression. A/A and B/B delivered texts differed in all ten comparisons; calibration tables varied. Four-client cache reuse was 0 tokens in A1/B1 and 94 in B2/A2. No matched-input first numerical divergence has been located. Static review found no concrete eligibility, missing-job, duplicate-job or overlapping-write defect, so no speculative product fix was added. Timing-fed adaptive CPU/GPU dispatch remains a possible mechanism, not an established cause.

The window completed and restored production in 28.44 minutes, with maximum observed temperature 62°C. Independent read-only verification confirmed the original production binary/configuration/unit, 144 threads, four slots, original 12,24,35 boundaries, 255 W per GPU, authentication and listener ownership; the test port was released and experimental units exited. Restoration verification sent zero generation requests.

Checks and remaining work

• Full clean-branch engine built and ran on sm_89.
• Added host/native regression tests passed on the tested configurations.
• Same-binary, single-switch ABBA completed; all performance and quality samples retained.
• Bounded 40-answer independent quality follow-up, blind scoring and restoration verification completed; target gate failed.
• Predeclared whole-model answer-quality acceptance.
• Independent replication sufficient for promotion beyond the measured short workload.
• HIP / other GPU architecture validation.

Production is restored; this PR was neither merged nor promoted. The bounded failure inspection and independent quality follow-up are complete. No further model/performance run is planned in this follow-up; any future diagnostic needs a new, specific numerical or source-level reason. Keep this PR in Draft and the switch off by default.

Repository setting and historical experiment notes predate this update; the current build and acceptance status is recorded above.