Claude /skills exploration

[thad0ctor]

Inspired by Liger's liger-kernel-dev Claude skill (commit).
The reusable pattern: analyze input → structured profile → generate known files → hard validation gate.

Underlying Logic: Good skill candidates are tasks that (a) touch the same handful of files every time and (b) have a
correctness check expressible as concrete commands or a generated test.

---

1. Liger upstream-sync

Problem: integrations/liger/plugin.py reconciles upstream MODEL_TYPE_TO_APPLY_LIGER_FN against a
hand-maintained override set (axolotl_override_liger_fn, line ~85) and a long elif model_config_type ==
chain of bespoke patches (qwen3_5, gemma4, jamba, deepseek_v2, llama4…). When Liger ships native support for
a type Axolotl still hand-patches, CI stays green while the wrong kernel set is applied — a silent
correctness/perf regression, not a version error. (Seen for real in the liger-0.8.0 bump: qwen3_5 /
qwen3_5_moe / gemma4_text override shadowing.)

Skill: diff installed MODEL_TYPE_TO_APPLY_LIGER_FN against the elif chain + override set; report
"upstream now natively supports X, Axolotl still hand-patches it" and "upstream changed the
apply_liger_kernel_to_X signature (stale swiglu/glu param check)."

Validation gate: for each covered model_config_type, assert which path fires (native vs override) and
that the expected modules (RMSNorm, SwiGLUMLP, FLCE forward) were actually swapped.

---

2. Prompt-strategy / dataset-format authoring

Problem: Most repetitive contributor task. type: → module via importlib in
prompt_strategies/__init__.py. Every new format touches the same 3 places: a module in
prompt_strategies/, a golden test in tests/prompt_strategies/, a doc page in docs/dataset-formats/.

Skill: scaffold the strategy module, register the type:, generate the doc page, and emit a golden
round-trip test.

Validation gate: axolotl preprocess --debug for label-masking inspection + a generated test asserting
exact token IDs and the masked-label span (the loss-masking bug reviewers can't eyeball).

---

3. Integration / plugin scaffold

Problem: BasePlugin exposes ~20 lifecycle hooks (pre_model_load, post_model_build,
get_input_args, get_trainer_cls, create_optimizer…). Contributors silently skip hooks.

Skill: scaffold integrations/<name>/{__init__.py, args.py, README.md}, wire get_input_args, add a
config example + load/parse smoke test.

Validation gate: plugin loads, args parse, lifecycle hooks no-op cleanly.

---

4. Config-field addition

Problem: Bounded but easy to under-test in a config-driven project. docs/config-reference is
auto-generated (docs/scripts/generate_config_docs.py), so the surface is: Pydantic schema in
utils/schemas/, validator, regen docs, test.

Skill: add the field + validator, regenerate docs, scaffold a schema test.

Validation gate: schema round-trips, validator rejects bad combos, docs regenerate clean.

---

5. "Patch-retirement" check

Problem: Bumps are one-line + CI, but CI won't tell you which entries in monkeypatch/ and integration
shims are now redundant because upstream transformers/trl/peft caught up.

Skill: scan monkeypatch/ + shims, flag candidate-removable patches after a bump.

Validation gate: for each flagged patch, confirm upstream now provides equivalent behavior (test passes
with the patch disabled). Note: largely subsumed by #1 for the Liger elif chain.

---

6. Chat-template generator

Problem: tests/prompt_strategies/ already has a large chat-template suite (test_chat_templates*.py —
thinking, tool-calls, mistral…). Adding a new template repeats an established pattern.

Skill: add a new chat template + auto-generate its rendered + tokenized snapshot test.

Validation gate: snapshot matches; tool-call / thinking / system-prompt variants render correctly.
Low-risk, high-acceptance.

---

7. Model verification harness

Problem: Threads the maintainer's needle — the bespoke part of model support (attention/RoPE/mask
patches) can't be one skill, but verifying a model can.

Skill: given a model, scaffold tiny LoRA + full configs against a tiny variant, run preprocess + a few
train steps.

Validation gate: loss is sane, sample-packing behaves, Liger/CCE compat flags resolve correctly. A
reproducible smoke gate for model PRs.

---**

✔️ Solution

Was chatting with @NanoCode012 about whether any skills are worth adding getting inspiration from Liger

❓ Alternatives

There are other dev/testing oriented skills consistent across pytorch/sglang/nemo and others that are valid but they would be lesser value:

Training-run diagnostic . The single most common pattern across all four repos: NeMo debug-training-logs, SGLang debug-distributed-hang + debug-cuda-crash, PyTorch distributed-triage + pt2-bug-basher. Axolotl already has the source material (docs/training_stability.qmd, docs/debugging.qmd); the skill turns that prose into a guided "loss is NaN / OOM / hang / loss not decreasing → diagnose" workflow. Glue over existing docs + commands, no new logic.

Issue → minimal-repro triage. PyTorch triaging-issues/scrub-issue/fix-issue, Megatron respond-to-issue. Take a user's failing config, produce a minimal repro (tiny model + tiny dataset, preserving the SFT/DPO/GRPO/LoRA/multimodal path).

📝 Additional Context

No response

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this feature has not been requested yet.
• I have provided enough information for the maintainers to understand and evaluate this request.

[NanoCode012]

IMO, sounds like (1) + 5 sounds good. 6 should usually be done when adding model either way, maybe a "model-checklist" skill which lists min TODOs (sounds like a simple doc is enough)?

(7) would be powerful.

All of these are dev only, is the standard to just include in base repo? Would it pollute regular user runs?

Issue → minimal-repro triage

This part is tricky. We usually want users to give us their configs as we don't have time to guess their specific device + env + config given all the knobs we offer..

Training-run diagnostic .

This could be a useful user skill but I'm not sure how good it would be compared to what the model would've seen from error log + online search + lookup with current user docs.

[thad0ctor]

IMO, sounds like (1) + 5 sounds good. 6 should usually be done when adding model either way, maybe a "model-checklist" skill which lists min TODOs (sounds like a simple doc is enough)?

(7) would be powerful.

All of these are dev only, is the standard to just include in base repo? Would it pollute regular user runs?

Issue → minimal-repro triage

This part is tricky. We usually want users to give us their configs as we don't have time to guess their specific device + env + config given all the knobs we offer..

Training-run diagnostic .

This could be a useful user skill but I'm not sure how good it would be compared to what the model would've seen from error log + online search + lookup with current user de hocs.

@NanoCode012 thanks!

I already have # 1 built locally and working, I'll submit a PR. This should be a nice base (the PR is the skills framework/documentation + 1) to then start working on 5.

I can also look at how to capture other dependency updates - the issue here is it would be best informed by some type of weekly upstream dep check workflow that creates an automated issue of potential bumps to the pyproject.toml - otherwise it becomes a scavenger hunt. I can PR this basic flow if you'd like but I want want your OK first because that's is changing how your repo is structured beyond just code changes. It would need to account for pip/uv compatibility, build wheels, etc so it gets trickier if you expand it repo-wide so maybe just key deps like torch, peft, bnb, attention kernels, etc. - worst case scenario it just generates an issue/report... you could have the agent create an analysis of the upstream updates (fixes/performance improvements/compatibility) in the issue to prioritize bumps conceivably

Model implementation checklist would be a good idea. I'll think this through

#7 will be a bit trickier given all the variables. maybe a parsing engine that resolves whether it is MOE, vision, etc. and diverges to various configs whichs iterates on config settings to compile a verification matrix, I could see this as being quite lengthy albeit consistent way to approach model compatibility and a support matrix per model of compatible features

Agreed on the triage, user reports are inconsistent at best... this is something I saw on large repos like pytorch (iirc) but doesn't really make sense here and 99% of people training models are likely already putting their logs into LLMs anyways to troubleshoot - it could provide more structured diagnosis though to assist power users and deduce common issues but for marginal value IMO

[thad0ctor]

All of these are dev only, is the standard to just include in base repo? Would it pollute regular user runs?

It seems pretty standard to include them in the repo (liger, pytorch, nemo, etc.) and are in a hidden file. From the other repos they all seem tailored to devs and contributors.

I'm not an expert on skills ... I from what I gather a basic summary of what they do and when they need to be used is put in the system prompt (minor context tax) but they really only get activated when certain context triggers set them off. Like any AI prompting, if the agent goes off on a tangent someone can divert it... I can look at whether there is some type of /command to enable or disable skills.