Speed up educational BPE training

[Marchematics]

Summary

Maintain BPE pair counts incrementally during training instead of rescanning the corpus for every merge.

This keeps the existing merge order and makes large vocabulary training practical.

Fixes #299

Tests

• tests/test_educational.py
• 1,000 randomized equivalence cases against the reference trainer
• ruff check and python -m py_compile

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Marchematics]

Benchmark on 1,000 random 80-character words (500 merges): 4.79s -> 0.27s (17.8x).

The merge vocabulary matches the original trainer across 1,000 randomized cases.

[sylvesterkaczmarek]

Could we make the randomized equivalence validation mentioned in the PR reproducible in the committed test suite?

This replaces the simple corpus rescan with linked nodes, occurrence sets, and lazy heaps, where tie-breaking and overlapping merges are fairly easy to regress. The committed diff currently adds only two deterministic cases.

A bounded fixed-seed differential test against the previous/reference trainer on small corpora and vocabulary sizes would give this optimisation a durable correctness oracle in CI. Since the PR specifically addresses training performance, one reproducible before/after benchmark would also help quantify the improvement and memory trade-off.

[Marchematics]

Added fixed-seed differential coverage against the reference trainer for six small corpora/vocabulary pairs, plus scripts/benchmark_educational.py for reproducible timing and peak-memory measurements. The benchmark reports 4.321s -> 0.239s on the committed workload.

[sylvesterkaczmarek]

Added fixed-seed differential coverage against the reference trainer for six small corpora/vocabulary pairs, plus scripts/benchmark_educational.py for reproducible timing and peak-memory measurements. The benchmark reports 4.321s -> 0.239s on the committed workload.

Thanks, this addresses my concern. The committed differential coverage and reproducible benchmark make the optimisation much easier to validate and maintain.