Skip to content

Commit d5884c1

Browse files
committed
docs: fix MatchArena stale description, note D independent of B, fix 50k→1k refs
1 parent 3f50bd5 commit d5884c1

2 files changed

Lines changed: 8 additions & 6 deletions

File tree

docs/architecture.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,9 @@ Implemented in `src/beam.rs`. One New pattern aligned against all Old patterns s
4848

4949
**Inter-pattern ordering constraint**: the first symbol of a new Old pattern must begin at a New position `>= max_covered_new`. Prevents mid-stream interleaving; does not detect full-sequence reorderings — see [docs/known-limitations.md](known-limitations.md).
5050

51-
**Match log / MatchArena**: beam search owns a `MatchArena` (flat `Vec<MatchNode>` linked list). Each `PartialAlignment` holds a single `u32` tail index rather than a cloned vec of match events. Forking copies one `u32`. After beam completion, `arena.collect(winning.log_tail)` walks the linked list once to reconstruct all match events for the winning alignment.
51+
**Match log / MatchArena**: beam search owns a `MatchArena` (flat `Vec<MatchNode>` linked list). Each `PartialAlignment` holds a single `u32` tail index. Forking copies one `u32`. After beam completion, `arena.collect(winning.log_tail)` walks the linked list once to reconstruct all match events for the winning alignment.
52+
53+
**`PartialAlignment` is allocation-free**: cursor state uses a `[u16; MAX_PATS]` fixed array (sentinel `u16::MAX` = not started) and a `[u64; 8]` bitmask for covered positions (512-symbol limit). Clone is a stack memcpy — no heap allocation in the beam inner loop.
5254

5355
**Why not pairwise**: the original implementation matched New against one Old pattern at a time and merged results. This is wrong — SPMA's compression gain comes from using multiple Old patterns to cover different spans of New simultaneously. Pairwise alignment misses cross-pattern coverage and systematically underestimates CD.
5456

docs/performance.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,10 @@
77
- **mimalloc**: global allocator replaced with mimalloc for parallel allocation throughput.
88
- **MDL cache**: total E cost cached during training beam passes — avoids a full extra pass to build `e_distribution`.
99
- **Infer match_log reuse**: level-0 `beam_search` result's `match_log` seeded directly into the N-level loop; level=1 extracts pid_seq from it without a second beam call.
10-
- **SymbolIndex on GrammarLevel**: inverted index `symbol_id → [(pattern_idx, pos)]` built once per level, eliminates per-call `HashMap` rebuild in `beam_search`. Measured gain on HDFS 50k/446k: <1% — bottleneck is beam candidate expansion, not index construction.
10+
- **SymbolIndex on GrammarLevel**: inverted index `symbol_id → [(pattern_idx, pos)]` built once per level, eliminates per-call `HashMap` rebuild in `beam_search`. Measured gain on HDFS 1k/446k: <1% — bottleneck is beam candidate expansion, not index construction.
1111

12-
Observed on HDFS (50k training corpus, 446k infer, release build, Apple Silicon):
13-
- Training (50k sequences): ~138s user, ~1m48s wall (sequential, 128% CPU)
12+
Observed on HDFS (1k training corpus, 446k infer, release build, Apple Silicon):
13+
- Training (1k sequences): ~138s user, ~1m48s wall (sequential, 128% CPU)
1414
- Infer (446k sequences, parallel, baseline): ~1816s user, ~123s wall (1480% CPU, 16 cores)
1515
- Infer (446k sequences, parallel, after H+I+J): ~637s user, ~42s wall (~2.85× user, ~2.9× wall)
1616

@@ -63,8 +63,8 @@ pub struct PatternStore {
6363
}
6464
```
6565

66-
**D — Parallel training**
67-
Training beam passes are currently sequential per level. Each new pattern is independent within a level — could parallelize with rayon, keeping grammar update sequential.
66+
**D — Parallel training** *(independent of B)*
67+
Training beam passes are currently sequential per level. Each new pattern is independent within a level — could parallelize with rayon, keeping grammar update sequential. Does not require the SoA refactor (B) — can be done on the current AoS layout.
6868

6969
**E — SIMD symbol comparison** *(after B)*
7070
AVX2: 8 `u32` symbols per instruction.

0 commit comments

Comments
 (0)