Add review-flydsl-kernel: legacy-spelling scanner distilled from 479 maintainer comments - #1047
Open
jhinpan wants to merge 3 commits into
Open
Add review-flydsl-kernel: legacy-spelling scanner distilled from 479 maintainer comments#1047jhinpan wants to merge 3 commits into
jhinpan wants to merge 3 commits into
Conversation
…view history
An agent reviewing a FlyDSL kernel PR currently has two rules for FlyDSL (in
aiter's review-pr) and no knowledge of what this repo's maintainers actually ask
for. This adds that, taken from the history rather than invented.
Corpus: all 865 PRs, of which 163 carry human review, 479 comments total.
coderfeli (273 comments over 101 PRs, Jan-Aug 2026) and sjfeng1999 (57 over 32)
dominate; the taste is consistent enough across eight months to encode. Every
rule cites the PRs the objection was actually made on.
Six families, ordered by how often they occurred:
F1 legacy spelling where the current one exists -- 46 of the 479 comments, by
far the largest. ir.*/ArithValue -> fx types, scf.If -> plain Python if,
buffer_ops -> fx.copy, SmemAllocator -> SharedAllocator, make_ptr ->
recast_iter. Ships as scan_legacy_spelling.py rather than prose: validated
against the files maintainers commented on, it flags 10 of 12; the two it
does not flag are cases where the pattern is not on an added line.
F2 a default argument that silently aliases another buffer (yanguahe #844 filed
three blockers on one PR for this; the same shape recurred in #629)
F3 duplicated code paths that should be shared
F4 config/dispatch/tile selection living inside the kernel file
F5 a new test the entry point CI runs cannot reach
F6 unvalidated cast at the C ABI boundary
It deliberately does not restate flydsl-kernel-authoring or kernel-code-cleanup,
does not duplicate review-pr's correctness rules, and does not judge performance.
F1 is a scanner and not a rule for a measured reason: across four controlled arms
on 14 PRs, a rule rewritten in prose caught 0 of 3 of the defects it targeted,
including after the instruction was moved to where the agent reliably executes it.
Five of the six families are removed before merge because a controlled test said they add nothing. Each was seeded into the real softmax backward kernel, opened as a PR, and reviewed twice -- once with review-pr alone, once with review-pr plus this skill. review-pr alone already caught four of the five: the default argument aliasing another buffer, the duplicated path, the config table inside the kernel file, and the unvalidated cast. Only the unreachable-test family needed this skill. Rules that change nothing are not free. They cost reviewer attention and dilute the findings that matter, and the output contract allows five findings per PR. What remains is F1, the legacy-spelling family: 46 of the 479 human review comments in this repo, shipped as scan_legacy_spelling.py, which flags 10 of the 12 files where a maintainer actually raised it. The seeded test is biased toward the control -- a 4-to-20 line diff whose only change is the defect is far easier to spot than the same pattern buried in a 2000-line PR -- so this is not proof the dropped families are worthless. It is that they have no measured benefit today. The corpus analysis is retained outside the repo and they can return with evidence; the file now states that standard for anything added.
jhinpan
changed the title
…ged an outcome Dropping F5 with the other four was wrong. It is the single family review-pr alone missed in the seeded test (4/5 -> 5/5), so it is the only one with direct evidence that it adds something, and it was cut for tidiness rather than for a reason. It returns as a scanner rather than prose, which is what the evidence supports. scan_unreachable_tests.py parses the test file, resolves what `__main__` reaches transitively, and reports only tests the diff ADDS. Validated three ways: - replaying #481 it names exactly the three tests coderfeli named (test_rmsnorm_fused_add and its dynamicquant/smoothquant variants -- "the fused/quant variants are not exercised in that path") - on a seeded PR adding one pytest-only test, it flags that test and nothing else - on a comment-only control it reports nothing Only added tests count. A first version reported absolute state and fired on 6 of 7 untouched tests in one file, because pytest-first files with a one-line __main__ are the norm here; that version would have been noise. F5 pairs with validate-kernel-pr from the other side: this catches a test that will not run, the shape grid catches shapes a running test never reaches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
An agent reviewing a FlyDSL kernel PR today has two FlyDSL-specific rules (both in aiter's
review-pr) and no knowledge of what this repo's maintainers actually ask for. Meanwhile the same objection gets typed by hand, over and over.Mined from all 865 PRs here: 163 carry human review, 479 comments total.
46 of those 479 comments are one objection: a legacy DSL spelling where the current one already exists. That is what this PR automates.
Technical Details
One rule and one scanner — deliberately.
scan_legacy_spelling.pycarries the pairs the maintainers ask for, andreview-prStep 1 runs it so the candidates are in context before the rule pass:ir.*/_mlir.*/ArithValue/ manual wrap-unwrapfx.Float32,fx.Int32)scf.IfOp/scf.ForOpif/forbuffer_ops.*fx.copy/ a copy atomarithwrapperfastmathhintSmemAllocatorSharedAllocatormake_ptrto retype a pointerrecast_itertiledCopy.partition_src/dstIt only looks at added lines, defaults to⚠️ , and groups all instances into one finding rather than spending five slots on style.
Test Plan
.pyfile where coderfeli or sjfeng1999 left a comment of this class, check it flags that file from the PR's diff.kernels/norm/softmax_bwd_kernel.py, open it as a PR in a fork, and review it twice — with and without this skill — keeping only rules that catch somethingreview-pralone does not. Plus a comment-only null control.Test Result
Scanner: 10 of 12 files flagged. Both misses were checked and are not scanner errors — in one the commented file is absent from the diff as fetched today, in the other the pattern is only on pre-existing lines, which the scanner ignores by design.
Five further families were tested and dropped. They were distilled from the same corpus and each cites real comments, but the seeded test said they add nothing:
review-pralone¹ the only one that changed an outcome, dropped with the rest for now: one rule's worth of benefit did not justify keeping a second, unvalidated section in the file. It is the first candidate to bring back.
No false-positive inflation: on the comment-only control the two arms produced 2 and 1 findings, both reasonable documentation notes rather than invented defects.
One control-arm run was discarded and re-run after it analysed a different PR's diff —
review-prStep 1 wrote to a fixed/tmp/pr.diffand two concurrent reviews overwrote each other. That is areview-prbug, fixed separately in ROCm/aiter#4870.Limitations
review-prandvalidate-kernel-prdo that.