diff --git a/flagscale_agent/skills/infer-env-setup/SKILL.md b/flagscale_agent/skills/infer-env-setup/SKILL.md index a5161b5..8241dbe 100644 --- a/flagscale_agent/skills/infer-env-setup/SKILL.md +++ b/flagscale_agent/skills/infer-env-setup/SKILL.md @@ -304,15 +304,34 @@ ssh "docker exec bash -c \ Record the version: `memory_write('_vllm_pinned_version', 'X.Y.Z')` -### Step 2: Install vLLM CPU-only +### Step 2: Install vLLM CPU-only (from source) + +Install from source to ensure `VLLM_TARGET_DEVICE=empty` takes effect correctly. +Pre-built PyPI wheels may bundle device-specific compiled extensions that conflict +with the plugin's hardware backend. ```bash +# Clone the pinned vLLM version ssh "docker exec bash -c \ - 'VLLM_TARGET_DEVICE=empty pip install vllm== \ + 'cd /workspace/adapt/-vllm- && \ + git clone https://github.com/vllm-project/vllm.git vllm-src && \ + cd vllm-src && git checkout v && \ + git log -1 --oneline'" + +# Install from source with empty device target +ssh "docker exec bash -c \ + 'cd /workspace/adapt/-vllm-/vllm-src && \ + VLLM_TARGET_DEVICE=empty pip install -e . \ --extra-index-url https://download.pytorch.org/whl/cpu \ 2>&1 | tee /workspace/adapt-logs/install_vllm.log'" ``` +> **Why source install?** `VLLM_TARGET_DEVICE=empty` must be set at *build time* when +> installing from source so that no CUDA/hardware C extensions are compiled. Installing +> a pre-built wheel from PyPI may silently include compiled ops that break on non-NVIDIA +> backends. Source install with this flag produces a pure-Python, device-agnostic vLLM +> that the plugin can override completely. + ### Step 3: Clone vllm-plugin-FL (fresh workspace) ```bash diff --git a/flagscale_agent/skills/infer-hw-adapt/SKILL.md b/flagscale_agent/skills/infer-hw-adapt/SKILL.md index 312e32c..7b51ad2 100644 --- a/flagscale_agent/skills/infer-hw-adapt/SKILL.md +++ b/flagscale_agent/skills/infer-hw-adapt/SKILL.md @@ -13,6 +13,11 @@ keywords: - metax - ascend - moore-threads +- musa +- iluvatar +- sunrise +- hygon +- ppu - plugin - pr requires: @@ -59,7 +64,7 @@ constraints: description: Every temporary workaround must have a TODO comment trigger: tools: [edit_file, write_file] - keywords: [workaround, temporary, hack, TODO] + keywords: [workaround, temporary, hack] prompt: Check if the workaround has a TODO stating when it can be removed correction: "Add `# TODO: Remove when ` above the workaround." - id: squash_before_pr @@ -81,17 +86,7 @@ context_injection: --- # Hardware Adaptation after Plugin Upgrade -Adapt and fix vllm-plugin-FL for specific hardware backends after each plugin version upgrade. - -## When to Use This Skill - -Every time vllm-plugin-FL upgrades its base vLLM version (e.g., 0.19 → 0.20), hardware-specific code paths may break because: -- vLLM internal APIs change (worker, model_runner, ops dispatch) -- New Triton kernels are introduced that the hardware's Triton backend doesn't support -- FlagGems op coverage may lag behind new vLLM requirements -- Plugin patch points may shift or become invalid - -This skill covers the **adaptation and testing cycle** for one hardware backend per invocation. Environment setup (SSH, container, installation) is handled by `infer-env-setup`. +Adapt and fix vllm-plugin-FL for specific hardware backends after each plugin version upgrade. When vLLM bumps its base version (e.g., 0.19 -> 0.20), hardware-specific code paths break because internal APIs shift, new Triton kernels are introduced, or FlagGems op coverage lags. ## Prerequisites @@ -101,24 +96,15 @@ Before starting adaptation, ensure the environment is ready (via `infer-env-setu - vLLM (CPU-only), vllm-plugin-FL (editable), and FlagGems installed - All imports verified (`import vllm`, `import vllm_fl`, `import flag_gems`) -If any of these are not ready, run `infer-env-setup` first. - ## Critical Rules -1. **Test in order**: unit → functional → offline inference → serving. Fix each stage before proceeding. -2. **Never modify vLLM source** — all hardware adaptations go through plugin patches. -3. **Stream and persist logs** — use `2>&1 | tee /workspace/adapt-logs/.log`; diagnose from log files, don't re-run commands. -4. **After tests pass, review all changes** — remove anything unnecessary before PR. -5. **One patch per failure** — fix one issue, re-test, then move to the next. -6. **Patches are hardware-gated** — use `if current_platform.is_()` or equivalent. -7. **Every workaround has a TODO** — state when it can be removed. -8. **Sync code before testing** — if editing locally, push changes to container before running tests. -9. **Check device occupancy before tests** — use the backend's monitoring tool to confirm devices are free. -10. **Use tmux for long-running commands** — SSH sessions will timeout otherwise. -11. **Workspace orientation first** — run Stage 0 probe before any test. Record plugin path, branch, vLLM version, and model path in memory. Never guess paths. -12. **Read once, memorize** — after reading any source file, record key findings with `memory_write`. Check `memory_read` before re-opening a file. Never read the same file twice unless it was modified. -13. **Squash before PR** — squash all adaptation commits into one clean commit with a comprehensive message. Remove debug tools before squashing. -14. **Batch independent tool calls** — when multiple shell commands, file reads, or memory operations are independent, execute them in one response. +1. **Test in order**: unit -> functional -> offline inference -> serving. Fix each stage before proceeding. +2. **Never modify vLLM source** -- all patches go through plugin files in `vllm_fl/`. +3. **Stream and persist logs** -- use `2>&1 | tee /workspace/adapt-logs/.log`. +4. **One patch per failure** -- fix, re-test, then move to the next. +5. **Patches are hardware-gated** -- use `if current_platform.is_()` or vendor_name check. +6. **Every workaround has a TODO** -- state when it can be removed. +7. **Squash before PR** -- squash all adaptation commits into one clean commit. --- @@ -132,35 +118,23 @@ Before any test, record all paths to memory to avoid path confusion: ```bash ssh "docker exec bash -c ' - echo \"=== plugin workspace ===\" && - find /workspace -name \"vllm_fl\" -type d 2>/dev/null | head -5 && - echo \"=== plugin git info ===\" && - for dir in \$(find /workspace -name \".git\" -type d 2>/dev/null | grep vllm-plugin-FL); do - repo=\$(dirname \$dir) - echo \"Repo: \$repo\" - git -C \$repo branch --show-current - git -C \$repo log -1 --oneline - done && - echo \"=== vllm version ===\" && + find /workspace -name vllm_fl -type d 2>/dev/null | head -5 && python3 -c \"import vllm; print(vllm.__version__)\" && - echo \"=== plugin installed ===\" && python3 -c \"import vllm_fl; print(vllm_fl.__file__)\" && - echo \"=== adapt-logs ===\" && ls -lh /workspace/adapt-logs/ 2>/dev/null | tail -10 && - echo \"=== models ===\" && ls -lh /workspace/models/ 2>/dev/null | head -10 '" ``` -**Immediately record to memory:** +Immediately record to memory: ``` memory_write('_plugin_workspace', '') -memory_write('_plugin_branch', '') memory_write('_vllm_version', '') memory_write('_model_path', '/workspace/models/') -memory_write('_log_dir', '/workspace/adapt-logs') ``` +**Never guess paths. Always read from memory or re-probe.** + ### Stage 1: Unit Tests ```bash @@ -185,7 +159,7 @@ ssh "docker exec bash -c ' '" ``` -Monitor with `duration=300`, `process_pattern="pytest"`, `fail_pattern="FAILED|ERROR|hang|timeout"`. Functional tests can hang on graph capture failures — if a test hangs beyond 5 min, kill it and diagnose with `-k `. +Monitor with `duration=300`, `process_pattern="pytest"`, `fail_pattern="FAILED|ERROR|hang|timeout"`. If a test hangs beyond 5 min, kill it and diagnose with `-k `. Purpose: verify operator correctness, kernel dispatch, dtype handling. @@ -200,33 +174,30 @@ ssh "docker exec bash -c ' '" ``` -Monitor with `duration=600`, `process_pattern="python"`, `success_pattern="Prompt.*Output:|Generated text:"`. Model loading takes 2–4 min on first run. +Monitor with `duration=600`, `process_pattern="python"`, `success_pattern="Prompt.*Output:|Generated text:"`. Model loading takes 2-4 min on first run. Purpose: full model execution without serving overhead. Validates model loading, forward pass, sampling. ### Stage 4: Serving Test ```bash -# Terminal 1 — start server +# Terminal 1: start server ssh "docker exec bash -c ' VLLM_PLUGINS=fl vllm serve /workspace/models/ \ --tensor-parallel-size 2 \ - --enforce-eager \ --trust-remote-code \ 2>&1 | tee /workspace/adapt-logs/serving_$(date +%Y%m%d_%H%M%S).log '" -# Terminal 2 — send request after server is ready -ssh "docker exec bash -c ' - curl -s http://localhost:8000/v1/completions \ - -H \"Content-Type: application/json\" \ - -d \"{\\\"model\\\": \\\"/workspace/models/\\\", - \\\"prompt\\\": \\\"Hello, world\\\", - \\\"max_tokens\\\": 20}\" -'" +# Terminal 2: test request (after "Uvicorn running on" appears) +ssh "docker exec curl -s http://localhost:8000/v1/completions \ + -H 'Content-Type: application/json' \ + -d '{\"model\": \"/workspace/models/\", \"prompt\": \"Hello\", \"max_tokens\": 20}'" ``` -Monitor serving with `success_pattern="Application startup complete"`, `duration=300`. +Monitor server log with `success_pattern="Uvicorn running on"`, `fail_pattern="Error|Traceback"`. + +Purpose: validate the full serving stack under real HTTP load. --- @@ -235,9 +206,9 @@ Monitor serving with `success_pattern="Application startup complete"`, `duration The key principle: **detect first, patch only what's actually missing in the installed version.** ```bash -# Check if an API exists before patching +# Check what API exists before patching ssh "docker exec python3 -c \ - 'from vllm.worker import Worker; print(dir(Worker))'" + 'import vllm.worker.worker as w; print(dir(w.Worker))'" ``` **Common breakage patterns and fixes:** @@ -249,226 +220,19 @@ ssh "docker exec python3 -c \ | FlagGems op missing | `NotImplementedError` for op | Add fallback or report to FlagGems upstream | | Attention backend change | Wrong attention output | Update `vllm_fl/attention/_attn.py` | | Model runner API shift | `TypeError` on runner call | Update `vllm_fl/model_runner/_runner.py` | +| Platform detection wrong | Device routed to wrong code path | Set `is_cuda_alike()`, `use_custom_allreduce()` correctly in `vllm_fl/platform.py` | +| Patch applied twice | Double-patched function crashes or loops | Add idempotency guard (`_patches_applied` flag) at top of `apply_patches()` | +| `torch.accelerator` API missing | `AttributeError` on `torch.accelerator.*` | Stub missing attrs in `patch.py` with backend-specific equivalents | +| Ops registry conflict | `c10::Error: duplicate op registration` | Pre-load vendor C extension before registering schemas in `_C_ops_registry.py` | +| CUDAGraph capture timeout | Server killed during graph capture | Set `VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS` to a large value (e.g., 7200) | +| FlagGems fast-path on wrong backend | Wrong output on non-CUDA hardware | Restrict FlagGems-routed ops to `current_platform.is_cuda()` only | +| New MoE dispatch point | `NotImplementedError: invoke_fused_moe_triton_kernel` | Register vendor MoE kernel in `vllm_fl/dispatch/backends/vendor//register_ops.py` | --- ## Copy-then-Patch Discipline -1. **Copy first**: `cp /.py /vllm_fl/.py` — verbatim, no edits -2. **Patch via targeted edits**: use `edit_file` for specific lines, not `write_file` for the whole file -3. **One category at a time**: group related changes, verify import, then continue -4. **Import check after each batch**: `python3 -c "from vllm_fl.X import Y; print('OK')"` -5. **Never rewrite from scratch**: if tempted to rewrite, stop and read more upstream code first - ---- - -## Stage 5: Clean-Up Before PR - -Review all changes before squashing: - -```bash -ssh "docker exec bash -c ' - cd /workspace/adapt/-vllm-/vllm-plugin-FL && - git diff main -'" -``` - -### Clean-Up Checklist - -- [ ] Every patch has a `# TODO: Remove when ...` comment -- [ ] No debug prints, temporary hacks, or commented-out code -- [ ] Patches are gated by `if current_platform.is_()` -- [ ] `git diff main` reviewed — only necessary changes remain -- [ ] No passwords, tokens, or API keys in code or comments -- [ ] No hardcoded IP addresses or internal hostnames - ---- - -## Stage 6: PR Submission - -### Squash commits - -```bash -# Count commits since branch point -git log --oneline main..HEAD | wc -l - -# Squash all into one -git rebase -i HEAD~ -# In editor: keep first as 'pick', change rest to 'squash' -``` - -### Commit message template - -``` -adapt(): vllm-plugin-FL compatibility for vLLM - -Backend: () -vLLM version: - -Changes: -- : -- : - -Test results: -- Unit tests: PASS ( tests) -- Functional tests: PASS ( tests) -- Offline inference: PASS (, TP=) -- Serving: PASS (throughput: tok/s) - -FlagGems missing ops (need native implementation): -- : currently falls back to PyTorch - -Generated with FlagScale-Agent -``` - -### PR description sections - -1. **Target backend and vLLM version** -2. **Reason for each patch** (what broke and why) -3. **Test results summary** (which stages pass, key metrics) -4. **FlagGems needed-ops list** (ops that currently fall back to PyTorch) -5. **Agent attribution footer** - ---- - -## Related Skills - -- `infer-env-setup` — environment setup (SSH, container, installation) -- `infer-model-adapt` — port a new model into vllm-plugin-FL -- `debug-strategy` — systematic debugging when tests fail repeatedly -- `ops-discipline` — shell safety and environment awareness -- `workspace-layout` — shared storage paths for models and artifacts - ---- -Related skills (load if needed): `debug-strategy`, `ops-discipline` -### Stage 0: Workspace Orientation (MANDATORY) - -Before any test, record all paths to memory to avoid path confusion: - -```bash -ssh "docker exec bash -c ' - echo \"=== plugin workspace ===\" && - find /workspace -name \"vllm_fl\" -type d 2>/dev/null | head -5 && - echo \"=== plugin git info ===\" && - for dir in \$(find /workspace -name \".git\" -type d 2>/dev/null | grep vllm-plugin-FL); do - repo=\$(dirname \$dir) - echo \"Repo: \$repo\" - git -C \$repo branch --show-current - git -C \$repo log -1 --oneline - done && - echo \"=== vllm version ===\" && - python3 -c \"import vllm; print(vllm.__version__)\" && - echo \"=== plugin installed ===\" && - python3 -c \"import vllm_fl; print(vllm_fl.__file__)\" && - echo \"=== adapt-logs ===\" && - ls -lh /workspace/adapt-logs/ 2>/dev/null | tail -10 && - echo \"=== models ===\" && - ls -lh /workspace/models/ 2>/dev/null | head -10 -'" -``` - -Immediately record to memory: -``` -memory_write('_plugin_workspace', '') -memory_write('_plugin_branch', '') -memory_write('_vllm_version', '') -memory_write('_model_path', '/workspace/models/') -memory_write('_log_dir', '/workspace/adapt-logs') -``` - -**Never guess paths. Always read from memory or re-probe.** - -### Stage 1: Unit Tests - -```bash -ssh "docker exec bash -c \ - 'cd && \ - VLLM_PLUGINS=fl pytest tests/unit_tests/ -x -v \ - 2>&1 | tee /workspace/adapt-logs/unit_$(date +%Y%m%d_%H%M%S).log'" -``` - -Monitor with `duration=120`, `process_pattern="pytest"`. Unit tests complete in under 60s. -If pytest dies the monitor returns immediately rather than waiting out the full timeout. - -**Purpose**: verify import compatibility, API surface, basic plugin registration. - -### Stage 2: Functional Tests - -```bash -ssh "docker exec bash -c \ - 'cd && \ - VLLM_PLUGINS=fl pytest tests/functional_tests/ -x -v \ - 2>&1 | tee /workspace/adapt-logs/functional_$(date +%Y%m%d_%H%M%S).log'" -``` - -Monitor with `duration=300`, `process_pattern="pytest"`, `fail_pattern="FAILED|ERROR|hang|timeout"`. -If a test hangs beyond 5 min, kill it and diagnose the specific test with `-k `. - -**Purpose**: verify operator correctness, kernel dispatch, dtype handling. - -### Stage 3: Offline Inference - -```bash -ssh "docker exec bash -c \ - 'cd && \ - VLLM_PLUGINS=fl MODEL_PATH= TP_SIZE=2 \ - python examples/_offline_inference.py \ - 2>&1 | tee /workspace/adapt-logs/offline_$(date +%Y%m%d_%H%M%S).log'" -``` - -Monitor with `duration=600`, `process_pattern="python"`, `success_pattern="Prompt.*Output:|Generated text:"`. -Model loading takes 2–4 min on first run. - -**Purpose**: full model execution without serving overhead. Validates model loading, forward pass, sampling. - -### Stage 4: Serving Test - -```bash -# Terminal 1: start server -ssh "docker exec bash -c \ - 'VLLM_PLUGINS=fl vllm serve \ - --tensor-parallel-size 2 \ - --enforce-eager \ - --trust-remote-code \ - 2>&1 | tee /workspace/adapt-logs/serving_$(date +%Y%m%d_%H%M%S).log'" - -# Terminal 2: test request (after server is ready) -ssh "docker exec curl -s http://localhost:8000/v1/completions \ - -H 'Content-Type: application/json' \ - -d '{\"model\": \"\", \"prompt\": \"Hello, world\", \"max_tokens\": 20}'" -``` - -Monitor server log with `success_pattern="Uvicorn running on"`, `fail_pattern="Error|Traceback"`. - -**Purpose**: validate the full serving stack under real HTTP load. - ---- - -## Version-Adaptive Patching - -The key principle: **detect first, patch only what's actually missing in the installed version.** - -```bash -# Check what API exists before patching -ssh "docker exec python3 -c \ - 'import vllm.worker.worker as w; print(dir(w.Worker))'" -``` - -Patch categories by typical vLLM upgrade impact: - -| Category | What breaks | Plugin patch location | -|---|---|---| -| Worker API | `determine_num_available_blocks` signature | `vllm_fl/worker/` | -| Model runner | `capture_model` / graph capture hooks | `vllm_fl/model_runner/` | -| Ops dispatch | New Triton ops without backend support | `vllm_fl/ops/` | -| Attention backend | KV cache layout changes | `vllm_fl/attention/` | -| Sampling | New sampler params | `vllm_fl/sampling/` | - ---- - -## Copy-then-Patch Discipline - -1. **Copy first**: `cp /.py /vllm_fl//.py` — verbatim, no edits +1. **Copy first**: `cp /.py /vllm_fl//.py` -- verbatim, no edits 2. **Patch via targeted edits**: use `edit_file` for specific lines, not `write_file` for the whole file 3. **One category at a time**: group related changes, verify import, then continue 4. **Import check after each batch**: `python3 -c "from vllm_fl.models.X import Y; print('OK')"` @@ -486,13 +250,13 @@ Before squashing commits and opening PR: - [ ] No temporary `import pdb; pdb.set_trace()` or similar - [ ] Every patch has a `# TODO: Remove when ...` comment - [ ] Patches are gated by platform check (`if current_platform.is_()`) -- [ ] `git diff main` reviewed — only necessary changes remain +- [ ] `git diff main` reviewed -- only necessary changes remain ### Sensitive Content - [ ] No passwords, tokens, or API keys in code or comments - [ ] No SSH config, private keys, or `.pem` file paths committed - [ ] No hardcoded IP addresses or internal hostnames -- [ ] Run `git diff main | grep -iE '(password|token|secret|pem|private_key|proxy)'` — should return nothing +- [ ] Run `git diff main | grep -iE '(password|token|secret|pem|private_key|proxy)'` -- should return nothing ### Commits & PR - [ ] All commits squashed into one clean commit @@ -506,63 +270,6 @@ Before squashing commits and opening PR: ### Squash commits -```bash -# Count commits since branching from main -git log --oneline main..HEAD | wc -l - -# Squash all into one -git rebase -i HEAD~ -# In the editor: keep first as 'pick', change rest to 'squash' -``` - -### Commit message template - -``` -adapt(): vllm-plugin-FL v hardware adaptation - -Backend: -vLLM version: - -Changes: -- : -- : - -Test results: -- Unit tests: PASS ( tests) -- Functional tests: PASS ( tests) -- Offline inference: PASS (, TP=2) -- Serving: PASS (throughput: X tok/s) - -FlagGems ops needing native impl (currently falling back to PyTorch): -- : needed for - -🤖 Generated with FlagScale-Agent -``` - -### Push and open PR - -```bash -git push -u origin adapt/-vllm- -``` - -Then open PR targeting `main` branch of `flagos-ai/vllm-plugin-FL`. - ---- - -## Related Skills - -- `infer-env-setup` — environment setup (SSH, container, installation) -- `infer-model-adapt` — port a new model into vllm-plugin-FL -- `debug-strategy` — systematic debugging when tests fail repeatedly -- `ops-discipline` — shell safety and environment awareness -- `workspace-layout` — shared storage paths for models and artifacts - ---- -Related skills (load if needed): `debug-strategy`, `ops-discipline` -## Stage 6: PR Submission - -### Squash commits - ```bash # Count your adaptation commits git -C log --oneline main..HEAD | wc -l @@ -590,13 +297,12 @@ adapt(-vllm-): hardware adaptation for on vLLM , TP=, throughput= tok/s -- Serving: PASS — , first-token latency=ms +- Offline inference: PASS -- , TP= +- Serving: PASS -- , TP=, throughput= tok/s ## FlagGems missing ops (for upstream) - op_name_1: falls back to PyTorch, needs native implementation -- op_name_2: ... --- Co-authored-by: FlagScale-Agent @@ -607,7 +313,6 @@ Co-authored-by: FlagScale-Agent ```bash git -C push origin adapt/-vllm- -u -# Then open PR via GitHub UI or gh CLI: gh pr create \ --title "adapt(): vLLM hardware adaptation" \ --body-file /tmp/pr_body.md \ @@ -618,11 +323,11 @@ gh pr create \ ## Related Skills -- `infer-env-setup` — environment setup (SSH, container, installation) -- `infer-model-adapt` — port a new model into vllm-plugin-FL -- `debug-strategy` — systematic debugging when tests fail repeatedly -- `ops-discipline` — shell safety and environment awareness -- `workspace-layout` — shared storage paths for models and artifacts +- `infer-env-setup` -- environment setup (SSH, container, installation) +- `infer-model-adapt` -- port a new model into vllm-plugin-FL +- `debug-strategy` -- systematic debugging when tests fail repeatedly +- `ops-discipline` -- shell safety and environment awareness +- `workspace-layout` -- shared storage paths for models and artifacts --- Related skills (load if needed): `debug-strategy`, `ops-discipline` diff --git a/flagscale_agent/skills/infer-hw-adapt/SUMMARY.md b/flagscale_agent/skills/infer-hw-adapt/SUMMARY.md index 7ae36ff..7cbf56f 100644 --- a/flagscale_agent/skills/infer-hw-adapt/SUMMARY.md +++ b/flagscale_agent/skills/infer-hw-adapt/SUMMARY.md @@ -2,17 +2,17 @@ Adapt and fix vllm-plugin-FL for specific hardware backends after plugin version upgrades. -**Load when**: a vllm-plugin-FL version upgrade breaks hardware-specific code paths (worker, model_runner, ops dispatch), or when adding hardware support to an existing plugin version. Run after `infer-env-setup` confirms the environment is ready. +**Load when**: a vllm-plugin-FL version upgrade breaks hardware-specific code paths (worker, model_runner, ops dispatch, platform detection), or when adding hardware support to an existing plugin version. Run after `infer-env-setup` confirms the environment is ready. -**Full cycle**: Stage 0 orientation → Stage 1 unit tests → Stage 2 functional tests → Stage 3 offline inference → Stage 4 serving → Stage 5 clean-up → Stage 6 PR. +**Full cycle**: Stage 0 orientation -> Stage 1 unit tests -> Stage 2 functional tests -> Stage 3 offline inference -> Stage 4 serving -> Stage 5 clean-up -> Stage 6 PR. **Key principles**: -- Test in strict order — fix all failures at each stage before proceeding -- Never modify vLLM source — all patches go through plugin -- One patch per failure — fix, re-test, then move to next +- Test in strict order -- fix all failures at each stage before proceeding +- Never modify vLLM source -- all patches go through plugin +- One patch per failure -- fix, re-test, then move to next - Patches are hardware-gated with `if current_platform.is_()` - Every workaround has a TODO comment stating when it can be removed - Stream and persist all logs to `/workspace/adapt-logs/` - Squash all commits before PR -**Constraints**: 14 hard rules covering test order, source isolation, log persistence, patch discipline, platform gating, and PR hygiene. +**Constraints**: 7 hard rules covering test order, source isolation, log persistence, patch discipline, platform gating, TODO requirements, and PR hygiene. diff --git a/flagscale_agent/skills/infer-precision-check/SKILL.md b/flagscale_agent/skills/infer-precision-check/SKILL.md new file mode 100644 index 0000000..94831b5 --- /dev/null +++ b/flagscale_agent/skills/infer-precision-check/SKILL.md @@ -0,0 +1,530 @@ +--- +name: infer-precision-check +description: > + Verify inference output precision for vllm-plugin-FL on any hardware backend. + Compares token-level outputs against NVIDIA + upstream vLLM ground truth. + Use this skill whenever a task requires a correctness gate: model porting, + hardware adaptation, plugin version upgrades, or regression detection after + any code change. Trigger when the user says "check precision", "verify + correctness", "compare outputs", "run E2E precision test", "precision + alignment", or "does the output match NVIDIA?". Works for text-only and + multimodal models; greedy decoding (temperature=0) is the standard mode. +keywords: +- inference +- precision +- correctness +- e2e +- ground-truth +- token-comparison +- regression +- vllm +- hardware +requires: +- infer-env-setup +suggests: +- infer-hw-adapt +- infer-model-adapt +- debug-strategy +constraints: +- id: greedy_only + description: Always use temperature=0 (greedy) for precision comparisons + trigger: + keywords: [temperature, sampling, top_p, top_k] + prompt: Check if the comparison uses greedy decoding + correction: > + Set temperature=0, top_p=1.0, top_k=-1 on both GT and target sides. + Any sampling randomness makes token-level comparison meaningless. +- id: gt_first + description: Always generate GT outputs before running target hardware + trigger: + keywords: [target hardware, target backend, run on] + prompt: Check if GT outputs have already been collected and saved + correction: > + Run the GT script on NVIDIA first, save outputs to a JSON file, + then run the target hardware script and compare. +- id: same_model_weights + description: GT and target must load identical model weights + trigger: + keywords: [model path, model_path, load model] + prompt: Check if both sides point to the same model weights + correction: > + Verify with md5sum on representative weight files, or confirm both sides + mount the same NFS path. Different weights make comparison meaningless. +- id: log_all_outputs + description: All inference outputs must be saved to files, never just printed + trigger: + tools: [shell] + keywords: [generate, llm.generate, vllm serve, curl] + prompt: Check if the command saves output to a log file + correction: > + Pipe to 2>&1 | tee /workspace/adapt-logs/precision__.log + and save token IDs to JSON for offline comparison. +- id: tp_parity + description: TP size must be identical on GT and target unless testing TP scaling + trigger: + keywords: [tensor_parallel, tp_size, --tensor-parallel-size] + prompt: Check if TP size matches between GT and target runs + correction: > + Use TP=1 for the initial precision gate. Only increase TP after TP=1 passes. +context_injection: + always: + - Critical Rules + - Comparison Protocol + by_tool: + shell: + - Environment Probe + - GT Collection Commands + - Target Collection Commands +--- + +# Inference Precision Check + +Standardized correctness gate for vllm-plugin-FL on any hardware backend. + +## When to Use This Skill + +| Checkpoint | Trigger | +|---|---| +| Model porting (Step 13 in infer-model-adapt) | After offline inference passes | +| Hardware adaptation (Stage 4 in infer-hw-adapt) | After functional tests pass | +| Plugin version upgrade | After unit + functional tests pass | +| Any suspicious output divergence | Immediately on user report | +| Before PR submission | Always — final correctness gate | + +--- + +## Critical Rules + +1. **GT = NVIDIA + upstream vLLM** — never use the target hardware as ground truth. +2. **Greedy decoding only** — temperature=0. One mismatch position tells you exactly where the bug is. +3. **Same weights, same tokenizer** — verify with file hash or shared NFS path. +4. **Save to JSON, compare offline** — save token IDs and decoded text to files; don't eyeball terminals. +5. **TP=1 first** — eliminate TP sharding as a variable. Scale up only after TP=1 passes. +6. **Report divergence position** — "first mismatch at token 3" is actionable; "outputs differ" is not. +7. **Log both sides** — persist GT and target logs to `/workspace/adapt-logs/` with timestamps. + +--- + +## Comparison Protocol + +### Tier 1 — Token ID Match (strict) + +Exact token ID match at every position. + +Required for: +- First 10 tokens of any prompt +- Models with fully deterministic attention (no Flash Attention variants) +- Baseline text-only prompts + +Pass criterion: **all token IDs identical**. + +### Tier 2 — Top-K Token Set Match (relaxed) + +The correct token appears in top-K logits even if greedy selection differs. + +Acceptable for: +- Tokens 11–30 when hardware FP16/BF16 accumulation order differs +- Models using Flash Attention (small numerical noise expected) + +Pass criterion: **correct token in top-5 candidates at every position**. + +### Tier 3 — Semantic Match (fallback) + +Human-readable output conveys the same meaning. + +Use only when: +- Tier 2 cannot be achieved due to hardware numerical limits +- Divergence is in filler tokens (punctuation, articles), not content tokens +- The model uses hardware-specific fused kernels with documented precision loss + +Pass criterion: **output judged semantically equivalent on all test prompts**. +Document tier level achieved in the PR description. + +--- + +## Step 0: Environment Probe + +Before any test, verify both sides are ready. + +```bash +# NVIDIA GT machine: +ssh "python3 -c 'import vllm; print(vllm.__version__)' && \ + nvidia-smi --query-gpu=name,memory.free --format=csv,noheader" + +# Target machine: +ssh "docker exec bash -c ' + python3 -c \"import vllm; print(vllm.__version__)\" && + python3 -c \"import vllm_fl; print(vllm_fl.__file__)\" && + ls -lh +'" +``` + +Record to memory if not already present: +``` +memory_write('_gt_ssh_host', '') +memory_write('_model_path', '/workspace/models/') +memory_write('_log_dir', '/workspace/adapt-logs') +``` + +Verify weight identity (if GT and target do not share NFS): +```bash +# On GT: +ssh "md5sum /config.json /tokenizer.json" +# On target: +ssh "docker exec md5sum /config.json /tokenizer.json" +# Hashes must match. +``` + +--- + +## Step 1: Prepare Test Prompts + +Use the standard 5-prompt set. For multimodal models, add 3 image prompts (Step 3). + +Save to `/workspace/adapt-logs/precision_prompts.json` on the target machine: + +```json +[ + "Paris is the capital of", + "The quick brown fox jumps over the", + "In machine learning, a transformer model", + "Once upon a time in a land far away", + "The speed of light in vacuum is approximately" +] +``` + +For chat models, wrap each prompt in the model's chat template before generating. +Use `tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)`. + +--- + +## Step 2: Collect Ground Truth (NVIDIA) + +Run on the NVIDIA GT machine. Save token IDs and decoded text to JSON. + +```python +# gt_collect.py — run on NVIDIA GT machine +import json, os +from vllm import LLM, SamplingParams + +MODEL_PATH = "" +PROMPTS_FILE = "/workspace/adapt-logs/precision_prompts.json" +OUTPUT_FILE = "/workspace/adapt-logs/precision_gt.json" +TP_SIZE = 1 # always start with TP=1 + +prompts = json.load(open(PROMPTS_FILE)) +llm = LLM(MODEL_PATH, tensor_parallel_size=TP_SIZE, enforce_eager=True) +params = SamplingParams(temperature=0, max_tokens=32, top_p=1.0) + +outputs = llm.generate(prompts, params) +results = [] +for req_out in outputs: + results.append({ + "prompt": req_out.prompt, + "token_ids": req_out.outputs[0].token_ids, + "text": req_out.outputs[0].text, + }) + +json.dump(results, open(OUTPUT_FILE, "w"), indent=2, ensure_ascii=False) +print(f"GT saved to {OUTPUT_FILE}") +``` + +```bash +# On GT machine: +ssh "cd /workspace && \ + python3 gt_collect.py \ + 2>&1 | tee /workspace/adapt-logs/precision_gt_$(date +%Y%m%d_%H%M%S).log" +``` + +Copy GT results to target machine if not on shared NFS: +```bash +scp :/workspace/adapt-logs/precision_gt.json \ + /tmp/precision_gt.json +ssh "docker cp /tmp/precision_gt.json \ + :/workspace/adapt-logs/precision_gt.json" +``` + +--- + +## Step 3: Collect Target Hardware Outputs + +Run on target machine inside the Docker container. + +```python +# target_collect.py — run inside container on target machine +import json, os +os.environ["VLLM_PLUGINS"] = "fl" + +from vllm import LLM, SamplingParams + +MODEL_PATH = "" +PROMPTS_FILE = "/workspace/adapt-logs/precision_prompts.json" +OUTPUT_FILE = "/workspace/adapt-logs/precision_target.json" +TP_SIZE = 1 # match GT TP size + +prompts = json.load(open(PROMPTS_FILE)) + +if __name__ == "__main__": + llm = LLM(MODEL_PATH, tensor_parallel_size=TP_SIZE, enforce_eager=True, + trust_remote_code=True) + params = SamplingParams(temperature=0, max_tokens=32, top_p=1.0) + + outputs = llm.generate(prompts, params) + results = [] + for req_out in outputs: + results.append({ + "prompt": req_out.prompt, + "token_ids": req_out.outputs[0].token_ids, + "text": req_out.outputs[0].text, + }) + + json.dump(results, open(OUTPUT_FILE, "w"), indent=2, ensure_ascii=False) + print(f"Target saved to {OUTPUT_FILE}") +``` + +```bash +ssh "docker exec bash -c ' + cd /workspace && + python3 target_collect.py \ + 2>&1 | tee /workspace/adapt-logs/precision_target_$(date +%Y%m%d_%H%M%S).log +'" +``` + +--- + +## Step 4: Compare Outputs + +Run the comparison script. This is the pass/fail gate. + +```python +# compare.py — run anywhere with both JSON files accessible +import json, sys + +gt_file = "/workspace/adapt-logs/precision_gt.json" +target_file = "/workspace/adapt-logs/precision_target.json" + +gt = json.load(open(gt_file)) +target = json.load(open(target_file)) + +assert len(gt) == len(target), "Prompt count mismatch — check both files" + +all_pass = True +for i, (g, t) in enumerate(zip(gt, target)): + g_ids = g["token_ids"] + t_ids = t["token_ids"] + min_len = min(len(g_ids), len(t_ids)) + + first_mismatch = None + for pos in range(min_len): + if g_ids[pos] != t_ids[pos]: + first_mismatch = pos + break + + status = "PASS" if first_mismatch is None else f"MISMATCH@token{first_mismatch}" + if first_mismatch is not None: + all_pass = False + + print(f"[{status}] Prompt {i}: {g['prompt'][:40]!r}") + print(f" GT: {g_ids[:16]}") + print(f" Target: {t_ids[:16]}") + if first_mismatch is not None: + print(f" GT text: {g['text'][:80]!r}") + print(f" Target text: {t['text'][:80]!r}") + +print() +print("=== RESULT:", "ALL PASS" if all_pass else "FAILED — see mismatches above") +sys.exit(0 if all_pass else 1) +``` + +```bash +ssh "docker exec python3 /workspace/compare.py \ + 2>&1 | tee /workspace/adapt-logs/precision_compare_$(date +%Y%m%d_%H%M%S).log" +``` + +--- + +## Step 5: Multimodal Precision Check (if applicable) + +Skip this step for text-only models. + +### 5a. Prepare image test inputs + +Use 3 fixed, publicly available images with unambiguous content. +Save image URLs or local paths to `/workspace/adapt-logs/precision_mm_prompts.json`: + +```json +[ + { + "prompt": "What is shown in this image?", + "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/280px-PNG_transparency_demonstration_1.png" + }, + { + "prompt": "Describe the scene.", + "image": "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg" + }, + { + "prompt": "What text appears in the image?", + "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/Culinary_fruits_front_view.jpg/320px-Culinary_fruits_front_view.jpg" + } +] +``` + +### 5b. Collect GT and target for multimodal + +Use the same gt_collect / target_collect pattern, replacing `prompts` with +`vllm.inputs.TextPrompt` + image data. The comparison script (Step 4) applies +unchanged — compare token IDs for the generated portion only. + +--- + +## Step 6: TP Scaling Check (optional but recommended) + +After TP=1 passes Tier 1, verify TP>1 does not introduce sharding errors. + +```bash +# Re-run Steps 2–4 with TP_SIZE= on both GT and target. +# Accepted outcome: Tier 1 or Tier 2 pass. +# TP sharding commonly causes token mismatch at position 1–3 due to +# attention output accumulation order differences across GPUs/NPUs. +``` + +Record the highest TP size tested and the tier achieved at each size. + +--- + +## Step 7: Serving Mode Check + +Verify precision holds when model is served via OpenAI-compatible API. + +```bash +# Start server on target: +ssh "docker exec -d bash -c ' + VLLM_PLUGINS=fl vllm serve \ + --tensor-parallel-size 1 --enforce-eager \ + --trust-remote-code --port 8122 \ + 2>&1 | tee /workspace/adapt-logs/precision_serve_$(date +%Y%m%d_%H%M%S).log +'" + +# Wait for server to be ready: +sleep 30 + +# Query with greedy params: +ssh "docker exec bash -c ' + curl -s http://localhost:8122/v1/completions \ + -H \"Content-Type: application/json\" \ + -d \"{ + \\\"model\\\": \\\"\\\", + \\\"prompt\\\": \\\"Paris is the capital of\\\", + \\\"max_tokens\\\": 32, + \\\"temperature\\\": 0, + \\\"top_p\\\": 1.0 + }\" | python3 -m json.tool +'" +``` + +Compare the `text` field in the response against GT text from Step 2. +Token-level comparison is not possible via the API; use Tier 3 (semantic match) +for serving mode. + +--- + +## Failure Diagnosis Guide + +When a mismatch is found, narrow the root cause systematically. + +### Mismatch at token 0–2 (embedding or first attention block) + +Most likely causes, in order: +1. **Weight loading error** — wrong tensor mapped to wrong layer. Check `load_weights` in the model file. +2. **Tokenizer mismatch** — different tokenizer version or chat template applied on only one side. + Verify: `tokenizer.encode("Paris is the capital of")` gives same IDs on GT and target. +3. **Attention op numerical error** — backend Flash Attention gives wrong values. + Fix: add `enforce_eager=True` and re-test. If eager passes, the kernel is at fault. + +### Mismatch at token 3–10 (residual stream accumulates error) + +Most likely causes: +1. **MLP kernel precision** — GEMM accumulation order differs on target hardware. + Fix: compare with `torch.float32` forced via `dtype=torch.float32` in both runs. +2. **Layernorm implementation** — different epsilon or fused vs. unfused. + Fix: check `RMSNorm` / `LayerNorm` dispatch in plugin; try forcing PyTorch fallback. +3. **Activation function** — `SiLU`, `GELU` variants differ across hardware libs. + +### Mismatch at token 10+ (late drift) + +Likely Tier 2 acceptable. Verify top-K membership: + +```python +# Add to target_collect.py to capture logprobs for comparison +params = SamplingParams(temperature=0, max_tokens=32, top_p=1.0, + logprobs=5) # capture top-5 logprobs +# token_ids from GT should appear in logprobs on target side +``` + +### All prompts mismatch (systematic failure) + +Check: +- `VLLM_PLUGINS=fl` set on target but not GT (or vice versa) +- Different model weights or tokenizer versions +- Different dtype (fp16 vs bf16) on the two sides + +--- + +## Pass/Fail Checklist + +Before recording the result: + +- [ ] GT outputs collected with temperature=0, TP=1 on NVIDIA +- [ ] Target outputs collected with temperature=0, TP=1 and `enforce_eager=True` +- [ ] compare.py ran to completion with no Python exceptions +- [ ] Tier achieved documented (Tier 1 / 2 / 3) +- [ ] First mismatch position recorded (or "no mismatch") +- [ ] Log files saved to `/workspace/adapt-logs/` with timestamps +- [ ] TP>1 check completed (if model will be deployed with TP>1) +- [ ] Serving mode check completed (if model will be served via API) + +--- + +## Result Recording Template + +Copy into `workspace_experiment.update_last_attempt(result=...)` or PR description: + +``` +## Precision Check Result + +Backend: +Model: +vLLM: +TP: + +### Tier achieved: Tier <1|2|3> + +| Prompt | First mismatch token | GT text (first 20 tok) | Target text (first 20 tok) | +|---|---|---|---| +| "Paris is the capital of" | none / @token N | ... | ... | +| "The quick brown fox..." | none / @token N | ... | ... | +| "In machine learning..." | none / @token N | ... | ... | +| "Once upon a time..." | none / @token N | ... | ... | +| "The speed of light..." | none / @token N | ... | ... | + +### Notes + + +### Log files +- GT: /workspace/adapt-logs/precision_gt_.log +- Target: /workspace/adapt-logs/precision_target_.log +- Compare: /workspace/adapt-logs/precision_compare_.log +``` + +--- + +## Related Skills + +- `infer-env-setup` — environment setup (SSH, container, installation) +- `infer-model-adapt` — port a new model into vllm-plugin-FL (calls this skill at Step 13) +- `infer-hw-adapt` — hardware backend adaptation (calls this skill at Stage 4) +- `debug-strategy` — systematic debugging when precision failures repeat +- `ops-discipline` — shell safety, log persistence, environment awareness + +--- +Related skills (load if needed): `debug-strategy`, `infer-hw-adapt`, `infer-model-adapt` diff --git a/flagscale_agent/skills/infer-precision-check/SUMMARY.md b/flagscale_agent/skills/infer-precision-check/SUMMARY.md new file mode 100644 index 0000000..d9d24ff --- /dev/null +++ b/flagscale_agent/skills/infer-precision-check/SUMMARY.md @@ -0,0 +1,20 @@ +# Infer-Precision-Check — Summary + +Verify inference output precision for vllm-plugin-FL on any hardware backend, comparing token-level outputs against NVIDIA + upstream vLLM ground truth. + +**Load when**: any task reaches a correctness gate — model porting (after offline inference passes), hardware adaptation (after functional tests pass), plugin version upgrades (after unit + functional tests pass), or any suspected output divergence. Also load before PR submission as the final correctness check. + +**Full cycle**: Step 0 env probe → Step 1 prepare prompts → Step 2 collect GT (NVIDIA) → Step 3 collect target outputs → Step 4 compare → Step 5 multimodal check (if applicable) → Step 6 TP scaling check → Step 7 serving mode check. + +**Key principles**: +- GT = NVIDIA + upstream vLLM — never use target hardware as ground truth +- Greedy decoding only (temperature=0) — sampling randomness invalidates comparison +- Same weights, same tokenizer — verify with md5sum or shared NFS path +- Save token IDs to JSON, compare offline — do not eyeball terminal output +- TP=1 first — eliminate sharding as a variable before scaling up +- Report first mismatch position — "token 3" is actionable; "outputs differ" is not +- Log both sides to `/workspace/adapt-logs/` with timestamps + +**Tiers**: Tier 1 (exact token ID match) → Tier 2 (correct token in top-5 logits) → Tier 3 (semantic equivalence). Document tier achieved in PR description. + +**Constraints**: 5 hard rules covering greedy decoding, GT-first ordering, weight identity, log persistence, and TP parity.