[Skill] kernel-regression: Detect kernel performance regressions between commits

[sunway513]

Skill

kernel-regression

Priority: P1 (High — regressions can silently degrade production inference)

Motivation

Kernel performance can regress silently when upstream changes modify Triton codegen, CK templates, or AITER's JIT/tuning infrastructure. Without automated regression detection, these issues are only caught when users report throughput drops. A skill that compares kernel performance between two commits with statistical significance testing would catch regressions before they ship.

What This Skill Should Do

1. Checkout and build: Build AITER at both the base commit and the test commit (or use pre-built wheels if available)
2. Run benchmark suite: Execute a standard benchmark suite (key GEMM shapes, attention configs, MoE shapes) on both builds
3. Statistical comparison: Compare results with confidence intervals — flag regressions only when the difference exceeds noise (e.g., >3% with p<0.05)
4. Generate regression report: Output a markdown report with: regressed kernels, magnitude of regression, affected shapes, and suspected cause (based on git diff)
5. Bisect support: If a regression is confirmed, suggest the commit range to bisect based on which files changed

Acceptance Criteria

• Can compare kernel performance between two git refs (commits, branches, tags)
• Uses statistical significance testing (not just single-run comparison)
• Flags regressions exceeding a configurable threshold (default 3%)
• Covers GEMM (FP8, FP4, BF16), attention (FA, MLA), and MoE kernels
• Produces a structured markdown report with regression details
• Suggests bisect range based on changed files