[Skill] jit-debug: Diagnose JIT compilation failures

[sunway513]

Skill

jit-debug

Priority: P0 (Critical — JIT failures block all kernel execution)

Motivation

AITER's JIT system compiles HIP/Triton kernels on first use. Failures are common during development: missing modules, hipcc version mismatches, lock contention on multi-GPU nodes, and the SystemExit vs RuntimeError handling bug (where SystemExit in JIT crashes the entire process instead of falling back gracefully). Diagnosing these requires reading scattered logs, checking environment variables, and understanding the JIT cache/lock mechanism. A skill codifies this into a repeatable workflow.

What This Skill Should Do

1. Collect diagnostics: Gather hipcc version, ROCm version, GPU arch (gfx942/gfx950), JIT cache location, and environment variables (AITER_JIT_DIR, HIP_VISIBLE_DEVICES)
2. Analyze error logs: Parse JIT compilation errors, classify into categories (missing module, compilation error, lock timeout, permission error, architecture mismatch)
3. Check common issues: Verify hipcc is on PATH, check for stale .lock files in JIT cache, detect SystemExit vs RuntimeError handling in jit/core.py
4. Suggest fixes: Provide targeted fix recommendations based on error category (clear cache, update hipcc, fix error handling, etc.)
5. Validate fix: Re-run the failing JIT compilation to confirm the fix works

Acceptance Criteria

• Can diagnose hipcc version mismatch errors
• Can detect and clean stale JIT lock files
• Can identify SystemExit vs RuntimeError handling issues in jit/core.py
• Can detect missing module compilation errors and suggest which modules need building
• Can handle multi-GPU lock contention scenarios
• Produces a structured diagnostic report with error category and recommended fix