Skip to content

Commit 344d577

Browse files
Teq2412claude
andcommitted
docs(roadmap): require end-to-end smoke test before landing slime data-contract PR
Unit tests pin the _sample_to_payload contract in isolation, but an integration bug would still hit the first user. Add a required pre-PR smoke-test gate to the slime-data-contract plan: regenerate the JSONL, run train.sh with NUM_ROLLOUT=10 on Qwen2.5-3B-Instruct, confirm rollout metrics and training steps, paste evidence in the PR body. Also adds a unit-test item for the shallow-copy invariant (mutations to the returned payload must not leak into Sample.metadata). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 4c92007 commit 344d577

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

docs/roadmap/committed/slime-data-contract.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -450,6 +450,27 @@ Recommend (2) for one release, then remove in the next minor.
450450
`{"task_id": "x"}`.
451451
- [ ] Unit: `_sample_to_payload(Sample(metadata={}))``{}`.
452452
- [ ] Unit: `_sample_to_payload(Sample())` (no metadata attr) → `{}`.
453+
- [ ] Unit: returned dict is a shallow copy — mutating it does not
454+
leak into `Sample.metadata`.
455+
456+
**Required pre-PR smoke test**
457+
458+
Unit tests cover the contract in isolation; they can't catch an
459+
integration bug that would hit users on the first run. Before
460+
opening the PR, run an end-to-end training smoke test against a
461+
real ACR deployment and paste the evidence into the PR body:
462+
463+
- [ ] Regenerate `gsm8k_tiny.jsonl` with the new `{prompt, metadata}`
464+
shape.
465+
- [ ] Run `bash train.sh` with `NUM_ROLLOUT=10` on Qwen2.5-3B-Instruct.
466+
- [ ] Confirm:
467+
- at least one rollout step completes (`rollout/raw_reward` is
468+
non-trivial, i.e. > 0)
469+
- `train/loss`, `train/grad_norm`, `train/step` appear in logs
470+
- no `Exception`, `Traceback`, or `FAILED` session status
471+
- [ ] Capture the rollout-0 metrics line + a few sample rewards in
472+
the PR description so reviewers can see the contract worked
473+
end-to-end.
453474

454475
**Release**
455476

0 commit comments

Comments
 (0)