[Skill] triton-vs-asm: Auto-compare Triton and ASM backend performance

[sunway513]

Skill

triton-vs-asm

Priority: P1 (High — backend selection directly impacts inference throughput)

Motivation

AITER supports multiple backends (ASM, CK-Tile, Triton, hipBLASLt) for the same operators. Performance varies significantly by shape — ASM kernels dominate decode (M=1) while Triton/hipBLASLt may win on larger batch sizes. Currently, comparing backends requires manual benchmarking with ad-hoc scripts. A skill automates this comparison across shape lists, producing win-rate tables that feed directly into tuning CSVs.

What This Skill Should Do

1. Parse shape lists: Accept shape lists from CSV files (e.g., inference_gemm_pretune.csv) or command-line M,N,K specifications
2. Run benchmarks: Execute both Triton and ASM backends for each shape with proper warmup and timing (median of N runs)
3. Compute win rates: Calculate per-backend win rate, average speedup, and identify crossover points (shapes where leadership changes)
4. Generate tables: Output markdown win-rate summary table and detailed per-shape CSV with columns: M, N, K, Triton_TFLOPS, ASM_TFLOPS, Winner, Speedup
5. Identify patterns: Highlight shape ranges where each backend excels (e.g., "ASM wins 95% for M<=4, Triton wins 60% for M>=64")

Acceptance Criteria

• Can benchmark FP8 GEMM (F8F8BBS) shapes across Triton and ASM backends
• Can benchmark FP4 GEMM shapes across Triton, ASM, and CK-Tile backends
• Produces per-shape comparison CSV with TFLOPS and winner columns
• Computes summary win-rate statistics with shape-range breakdowns
• Handles GPU allocation correctly (HIP_VISIBLE_DEVICES for multi-GPU nodes)
• Results are reproducible (proper warmup, median timing, statistical reporting)