|
| 1 | +--- |
| 2 | +id: agents |
| 3 | +title: Agents |
| 4 | +sidebar_position: 2 |
| 5 | +--- |
| 6 | + |
| 7 | +# AGENTS.md |
| 8 | + |
| 9 | +Project context + traps for future agents (Claude Code, Codex, Cursor, whatever) editing this repo. |
| 10 | + |
| 11 | +## What this project is |
| 12 | + |
| 13 | +LoRA + merged-bf16 fine-tune of `google/gemma-4-31B-it` to speak "caveman mode" natively. Style is defined by [JuliusBrussee/caveman](https://github.com/JuliusBrussee/caveman) (MIT). Status: shipped to HuggingFace as `JBrussee/gemma-4-31B-caveman` (bf16) and `JBrussee/gemma-4-31B-caveman-lora` (adapter). Trained May 17 2026 in ~50 min on a RunPod RTX PRO 6000 Blackwell 96GB. |
| 14 | + |
| 15 | +The user is **Julius Brussee** — GitHub `JuliusBrussee`, HF `JBrussee`. Note the two handles don't match. |
| 16 | + |
| 17 | +## What "caveman" means in code |
| 18 | + |
| 19 | +The trained model rewrites or answers in caveman style: drops articles (a/an/the), drops filler (just/really/basically/actually/simply), drops pleasantries and hedging, allows fragments, follows the pattern `[thing] [action] [reason]. [next step].`. Critical invariant: **code blocks, function names, error strings, CLI commands stay byte-exact**. |
| 20 | + |
| 21 | +You speak caveman in conversation with Julius. You DO NOT speak caveman in code comments, commits, PR descriptions, or security warnings. See [`feedback-caveman-mode`](https://github.com/JuliusBrussee/finetune-caveman) if memory is available. |
| 22 | + |
| 23 | +## Pipeline at a glance |
| 24 | + |
| 25 | +``` |
| 26 | +data/sources/*.py → data/build_corpus.py → data/synthesize.py → data/filter.py → data/split.py |
| 27 | + ↓ |
| 28 | + training/train_unsloth.py (Unsloth + TRL SFT, QLoRA NF4) |
| 29 | + ↓ |
| 30 | + eval/run_eval.py → eval/judge.py |
| 31 | + ↓ |
| 32 | + scripts/push_to_hub.py |
| 33 | +``` |
| 34 | + |
| 35 | +Every stage is **save-as-you-go + resume by key-hash**. Kill any step at any time and rerun — it picks up where it left off. This is mandatory because the synthesis step burns through CLI quotas (`claude -p`, `codex exec`) and you WILL hit a rate limit mid-3000-row run. |
| 36 | + |
| 37 | +## Specific traps (each cost real hours / $) |
| 38 | + |
| 39 | +These are real things that broke during the build of this model. They will probably re-bite if you regenerate or retrain. |
| 40 | + |
| 41 | +1. **TRL 0.17 API renames.** `SFTTrainer(tokenizer=...)` → `SFTTrainer(processing_class=...)`. `SFTConfig(assistant_only_loss=True)` does not exist on 0.17; only `completion_only_loss=True` is portable. The training script already handles this — don't revert. |
| 42 | + |
| 43 | +2. **Gemma 4 multimodal processor.** Unsloth's `FastLanguageModel.from_pretrained("google/gemma-4-...")` returns a `Gemma4Processor` (text+vision+audio), not a tokenizer. Unwrap with `tokenizer = getattr(tokenizer, "tokenizer", tokenizer)` before passing to TRL or before calling `apply_chat_template` with plain-string content. Already done in `train_unsloth.py`, `eval/run_eval.py`, `scripts/infer.py`. |
| 44 | + |
| 45 | +3. **`UNSLOTH_RETURN_LOGITS=1`.** Unsloth 2024.11+ returns empty logits by default. TRL's `compute_loss` needs real logits. Must set env var **before** `import unsloth`. `os.environ.setdefault` is unsafe — use unconditional assignment. Already pinned in `training/train_unsloth.py`. |
| 46 | + |
| 47 | +4. **`hf upload` is single-stream and gets throttled.** For folders > 10GB use `hf upload-large-folder --num-workers 8`. Same 62 GB upload finished in 5 min vs the naive tool's projected 6+ hours. |
| 48 | + |
| 49 | +5. **`bigcode/commitpackft` ships a script-based loader** that `datasets` v4+ refuses to run. Bypass: load the per-language `data.jsonl` URLs directly with `load_dataset("json", data_files=[urls], streaming=True)`. Already done in `data/sources/commitpack.py`. |
| 50 | + |
| 51 | +6. **`ronantakizawa/github-codereview` column names** are `reviewer_comment` + `diff_context` (not `comment` + `diff_hunk`). p25 of `quality_score` ≈ 0.36, so filters at 0.4+ kill 25%+ of rows. |
| 52 | + |
| 53 | +7. **CLI flag parsing on diff content.** Passing patch text containing `---` to `claude -p ---xyz` makes the CLI treat `---xyz` as an unknown flag and exit 1 with empty stderr. Always pipe content via stdin (`subprocess.run(cmd, input=content, ...)`), never as a positional arg. Already fixed in `data/synthesize.py`. |
| 54 | + |
| 55 | +8. **HF datasets streaming hangs on process exit.** Background connection threads keep Python alive for minutes after the main loop ends. `data/build_corpus.py` calls `os._exit(0)` at the end to force exit. Don't remove that. |
| 56 | + |
| 57 | +9. **Concurrent `claude -p` / `codex exec` rate-limits.** Each invocation incurs ~24k tokens of "session bootstrap" load. 8 concurrent workers blew Julius's Pro/Max budget in 10 min. Default in `data/synthesize.py` is now `--workers 3`. Codex also has a sliding-window burst limit; bursts of 100+ calls trigger a 6-hour cooldown. |
| 58 | + |
| 59 | +10. **RunPod / Ubuntu 24.04 PEP 668.** Pip refuses system-level installs by default. `training/runpod_bootstrap.sh` uses `--break-system-packages` since the pod is disposable. Don't replace that with a venv unless you want to also rewrite the bootstrap. |
| 60 | + |
| 61 | +11. **`huggingface-cli login` is deprecated.** Use `HF_TOKEN` env var only. The old command exits non-zero on the deprecation path, killing any `set -e` bash wrapper. |
| 62 | + |
| 63 | +12. **tmux sessions die silently** when the script under them exits non-zero with `set -e`. After kicking off a long bg job, always `sleep 3 && tmux ls` to verify it's still alive. Don't trust the launch echo. |
| 64 | + |
| 65 | +## How to extend the dataset (without retraining) |
| 66 | + |
| 67 | +If you want to add more pairs: |
| 68 | + |
| 69 | +```bash |
| 70 | +# Add or change a loader in data/sources/<name>.py implementing iter_records(limit) |
| 71 | +# yielding {prompt, source_normal?, source_seed?, category, origin, license}. |
| 72 | + |
| 73 | +# Register in data/build_corpus.py's DEFAULT_QUOTAS. |
| 74 | + |
| 75 | +# Run only that source: |
| 76 | +uv run python data/build_corpus.py --only <name> --limit 400 |
| 77 | +uv run python data/synthesize.py --backend claude --workers 3 # resumes |
| 78 | +uv run python data/filter.py --in data/out/raw_pairs.jsonl --out data/out/clean_pairs.jsonl |
| 79 | +uv run python data/split.py --in data/out/clean_pairs.jsonl |
| 80 | +``` |
| 81 | + |
| 82 | +Then retrain with `training/train_unsloth.py --config training/config.toml`. `resume_from_checkpoint=True` is auto-detected via `out_dir.glob("checkpoint-*")`. |
| 83 | + |
| 84 | +## How to tighten compression |
| 85 | + |
| 86 | +Current model compresses ~10-40% rather than the gold 50-70%. Filter upper bound in `data/filter.py` is `1.00` — relax to e.g. `0.75`, regenerate filtered set, retrain. Expect to lose 30-50% of pairs but gain harder compression. |
| 87 | + |
| 88 | +## How to add a new model variant (e.g. Gemma 3 4B) |
| 89 | + |
| 90 | +Change `training/config.toml`'s `model.base` and `model.max_seq_length`. Confirm Unsloth supports the variant (check https://unsloth.ai/docs). Smaller models can use fewer epochs and higher learning rate (e.g. 5e-4). Keep `completion_only_loss=True`. |
| 91 | + |
| 92 | +## Important external references |
| 93 | + |
| 94 | +- Style ruleset: https://github.com/JuliusBrussee/caveman/blob/main/skills/caveman/SKILL.md |
| 95 | +- Aligned baseline↔caveman seeds (10 pairs): https://raw.githubusercontent.com/JuliusBrussee/caveman/main/evals/snapshots/results.json |
| 96 | +- Unsloth docs: https://unsloth.ai/docs |
| 97 | +- TRL docs (mind version pinning): https://huggingface.co/docs/trl |
| 98 | +- Gemma 4 docs: https://ai.google.dev/gemma |
| 99 | + |
| 100 | +## Plan file from initial design |
| 101 | + |
| 102 | +`/Users/julb/.claude/plans/jaunty-doodling-treehouse.md` holds the original plan and v2 update for posterity. Decisions captured there: |
| 103 | +- Pick 31B Dense, skip the 26B-A4B MoE (Unsloth bnb-4bit doesn't support MoE fused 3D expert tensors) |
| 104 | +- QLoRA NF4 + bf16 compute, rank 16, all linear targets |
| 105 | +- Workflow-rich data over short Q&A (1500 workflow + 1500 qa, mixed) |
| 106 | +- Per-category compression bands |
| 107 | +- Code-fence integrity is a hard filter and an eval gate |
| 108 | + |
| 109 | +## Don't |
| 110 | + |
| 111 | +- Don't store API keys in this repo. They've been leaked once in chat history (the WANDB key Julius pasted, the HF token I echoed via `cat env.sh`). Both should be rotated; never write either to a tracked file. |
| 112 | +- Don't merge in untracked sibling directories (`blackjack/`, `crm/`, `smtp-microservice/`) — they're unrelated noise from Julius's working dir. |
| 113 | +- Don't commit anything under `data/out/`, `artifacts/`. Already gitignored. |
| 114 | + |
| 115 | +## Want to ship a new model? |
| 116 | + |
| 117 | +1. Bump version in pyproject.toml and HF repo name (e.g. `gemma-4-31B-caveman-v2`). |
| 118 | +2. Retrain. Push to a NEW HF repo, don't overwrite v1 (Julius cares about reproducibility). |
| 119 | +3. Update `README.md`'s eval table. |
| 120 | +4. Tag a release in GitHub. |
0 commit comments