Make default checkpoint retention conservative

[Helw150]

Closes #4456. Default keep for default_train/default_sft/default_dpo is now empty, so only the final checkpoint (plus rolling temp checkpoints for resumption) is retained.

Generated with Claude Code

[claude]

Claude finished @Helw150's task in 30s —— View job

---

I'll analyze this and get back to you.

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

View job run

[Helw150]

@claude Just sanity check me that we'll still keep doing rolling checkpoints for resumption with this change.

[claude]

Claude finished @Helw150's task in 25s —— View job

---

Sanity check: rolling resumption checkpoints

Yes — rolling temp checkpoints for resumption are preserved.

In experiments/defaults.py the CheckpointerConfig still sets save_interval=timedelta(minutes=10), and in Levanter's checkpointer (lib/levanter/src/levanter/checkpoint.py) save_interval and keep are independent:

• save_interval drives the time-based temporary checkpoint (is_temporary=True); the previous temp checkpoint is deleted after the next one is written — that's the rolling behavior (see ~lines 184–242).
• keep only controls permanent step-interval checkpoints. With the new _checkpoint_keep(None) -> [], no permanent intermediate checkpoints are written.
• The final end-of-training checkpoint is still saved as permanent regardless of keep.

So with this PR: every 10 minutes a single rolling temp checkpoint is maintained for preemption/resumption, and only the final checkpoint is retained permanently. Matches the PR description.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ebb85095e3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Disable default HF exports in SFT no-checkpoint mode

Setting steps_per_checkpoint to None here makes SFT look like it will keep only a final checkpoint, but steps_per_hf_export still defaults to 500, and default_sft forwards that value into default_train (experiments/defaults.py), which always gets an hf_save_path (lib/marin/src/marin/training/training.py) and registers periodic HF saves when hf_save_steps is non-null (lib/levanter/src/levanter/main/train_lm.py). In long SFT runs this still creates many permanent HF snapshots, so retention is not actually conservative and storage usage can grow unexpectedly.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Disable default HF exports in DPO no-checkpoint mode

This default now disables periodic Levanter checkpoints, but leaves steps_per_hf_export at 500, and default_dpo still resolves and passes that interval through (experiments/defaults.py), with HF save hooks enabled whenever hf_save_steps is set (lib/levanter/src/levanter/main/train_dpo.py). As a result, DPO runs with defaults continue to emit frequent permanent HF checkpoints, which undercuts the new conservative-retention behavior and can cause avoidable artifact bloat.

Useful? React with 👍 / 👎.