Skip to content

Commit 67316dc

Browse files
authored
Merge pull request #60 from actual-computer/toks/control-isolation
SPEC §3.6: control-token isolation certified per tokenizer; gen.c byte-domain chars of 7F..A0
2 parents 2ebc390 + 170702e commit 67316dc

15 files changed

Lines changed: 78441 additions & 71247 deletions

File tree

‎docs/algorithms/reach.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
Structural control-token isolation (SPEC §3.6)
2+
===============================================
3+
4+
src/core/reach.c decides at load, per tokenizer file, whether any added token's id can come out of
5+
`toks_encode(..., TOKS_ADDED_NONE)` for any input bytes. toks_info.control_isolation is 1 when it proves that none
6+
can, else 0. A consumer that places control tokens by position needs this. It encodes client text with no added-token
7+
recognition and puts the template's tokens in by id, as DeepSeek's API encoder does. Without the
8+
certificate, plain text could still spell a control id through the model itself.
9+
10+
§1 the claim §3 the piece test (gen.c toks_gen_piece)
11+
§2 the proof, per added id §4 the census
12+
13+
14+
1. the claim
15+
-------------
16+
17+
R is the ids of every added token in the file, special or not: harmony-format tokens and most families' `<think>` /
18+
`<tool_call>` are often special=false, and they are what a prompt template places. For all byte strings x, no id of R is
19+
in the NONE encode of x. The encode runs the normalizer, the pre-tokenizer chain and the model, and nothing else
20+
emits an id.
21+
22+
Analyzed: byte-level bpe, including the tiktoken readers. Sentencepiece-style bpe, unigram and wordpiece report 0, not
23+
analyzed: their models emit an unk or byte-fallback piece, and their vocabularies usually hold the added tokens.
24+
A 0 means "not proven", never "proven reachable".
25+
26+
27+
2. the proof, per added id
28+
---------------------------
29+
30+
Byte-level bpe emits only (a) the model's vocabulary ids, each an initial symbol (one byte's alphabet char) or a
31+
merge's product, (b) under `ignore_merges` (or tiktoken's ranks, `ids_as_rank`) the id whose string is a whole piece,
32+
and (c) the unk_token for a byte whose char the vocabulary lacks (config.c `drop_unk`). An id a of R is certified
33+
unreachable when every one of these is excluded:
34+
35+
- a is the dropped bytes' unk: reachable (input bytes are any bytes; gx-spark-x25 lacks 0xC0, 0xC1, 0xF5..0xFF).
36+
- a is not a vocabulary id (a >= n_vocab, or a tiktoken rank gap): the model never emits it.
37+
- a's string is not over the byte-level alphabet (decode-only, docs/breadth.md §1.4): bpe never forms it.
38+
- its string is one byte: an initial symbol, reachable. A merge makes it: reachable. Whether that merge can
39+
actually fire is not analyzed, which is sound and conservative.
40+
- otherwise, only (b) can emit it, so with neither ignore_merges nor ids_as_rank it is unreachable. With either,
41+
the question is whether its bytes b can be a final piece of the pre-tokenizer chain. A vocabulary with dropped
42+
bytes answers yes (run_text removes them from inside a piece). Otherwise §3 decides.
43+
44+
Normalization only narrows the texts that reach the pre-tokenizer, so a proof over all texts covers it.
45+
46+
47+
3. the piece test
48+
------------------
49+
50+
config.c records the chain's last splitting step (config.h `reach_*`). Every final piece is an element of that step,
51+
run over some text. Steps after it are ByteLevel without a regex, which splits nothing. If no step splits, every text
52+
is one piece and the answer is yes. If the last split is anything other than an Isolated, uninverted Split
53+
(Regex or String) or ByteLevel's gpt-2 regex, there is no proof and the answer is yes. Otherwise reach.c compiles that
54+
one step with gen.c's compiler, the template patterns included: their regex is the exact string. A ByteLevel before
55+
the step puts it in the byte domain. gen.c's toks_gen_piece then answers for b:
56+
57+
- a match: a piece equal to b needs a path of the step's program through exactly b's atoms (the VM's own atoms, an
58+
ill-formed byte as 0x110000 + byte). Assertions and lookaheads are relaxed: they may pass, since the text around
59+
the piece is anyone's. If no G_MATCH is reachable at b's end, b is no match in any context.
60+
- a gap: a gap holds no position where a match starts (the Pike VM adds a start thread at every position until it
61+
finds one, generic.md §5). Take a strict path from the start (no assertion, no lookahead) through one char c to
62+
G_MATCH. A match then starts at any position whose atom is c, whatever surrounds it. If b holds such an atom that
63+
needs no context (an ascii byte, or any byte in the byte domain), b is no gap.
64+
65+
Neither case means b is never a piece, so a is unreachable. Example: Nemotron's `<SPECIAL_12>` and `[INST]` under the
66+
o200k nemo regex. `[^\r\n\p{L}\p{N}]?[letters]+` stops at `_` and `]`, and ` ?[^\s\p{L}\p{N}]+[\r\n/]*` takes no
67+
letter, so no match spans the whole token. The second alternative matches `<` and `[` alone, strictly, so no gap holds
68+
them either.
69+
70+
71+
4. the census
72+
--------------
73+
74+
The tokenizer cache, 2026-10-27 (tests/c/test_reach.c pins the critical targets and gx-spark-x25):
75+
76+
certified (1) every byte-level bpe file but one: gpt-2, llama 3 / 3.1 / 3.2 / 4, GLM-5.3, Qwen 3 / 3.8,
77+
DeepSeek v3 / r1 / v3.1 / v3.2 / v4 / v4.1, o200k / gpt-oss, Mistral-Nemo, Nemotron 3 (4b, nano,
78+
omni), MiniMax text-01 / m1 / m2 / m3, ModernBERT, bloom, falcon, the generic-engine rows.
79+
not certified gx-spark-x25: its added `<unk>` is the unk of the bytes its vocabulary lacks, so `toks_encode("\xFF")`
80+
gives it (test_reach.c). Not analyzed: spm (llama 1 / 2, Mistral v0.1 to v0.3, gemma 1 to 4, phi-3,
81+
Yi, codellama), unigram, wordpiece.
82+
83+
The red proofs (tests/c/test_reach.c) are synthetic files where an added id is reachable: a merge's product, a
84+
one-byte token, a whole gpt-2 match under ignore_merges, any text under a ByteLevel without a regex, a gap of a
85+
letters-only split, and a dropped byte's unk. Each must be uncertified, and a concrete text's NONE encode must give
86+
the id. Mutants of each branch above (the drop unk, the byte and merge test, the match test, the gap test, the piece
87+
test as a whole) each fail at least one of them. Every certified file is also searched: each added token spelled in 15
88+
contexts, plus 2000 random byte strings, with no added id in a NONE encode.

‎docs/api-tests.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -355,7 +355,7 @@ Caller obligation (not counted): 280-281 "A push's out[0, cap) and the hold must
355355
| IN7b | TESTED | 328 | TOKS_PATH_NORMALIZE "normalizer on the compiled path" | test_e2e.c:211 (every pinned file: TOKS_PATH_NORMALIZE set; pinned so a change is a decision). Mutant receipt: api.c:776 without the bit survives master 245cc5c's make test (gb10a: GB10, A725 cores) and dies here |
356356
| IN8 | TESTED | 302 | cpu_features "the platform layer's feature bits" | test_api.c:897 (the load-time word, no probe after load), test_load.c:166 |
357357
| IN9 | TESTED | 303 | max_text "TOKS_MAX_TEXT" | test_api.c:896 |
358-
| IN10 | TESTED | 304 | control_isolation "1 when SPEC §3.6 is certified for this tokenizer" | test_e2e.c:211 (0 for every pinned file: nothing is certified in 0.3). Mutant receipt: api.c:779 made 1 survives master 245cc5c's make test (gb10a: GB10, A725 cores) and dies here. **Finding** (stays): the header could say it is 0 in 0.3 |
358+
| IN10 | TESTED | 304 | control_isolation "1 when SPEC §3.6 is certified for this tokenizer" | test_e2e.c:212 (since 0.3.3: 1 for every byte-level pin, 0 for the unigram pins) and test_reach.c (the proof's red files: each uncertified, with a text whose NONE encode gives the added id; the certified ones searched; the census pins). Mutants of reach.c's and gen.c's branches each fail test_reach (docs/algorithms/reach.md §4) |
359359
| IN11 | TESTED | 306 | source_sha256 | test_e2e.c:207 (14 files' sha-256 pinned); nightly python/tests/test_api.py:59 |
360360
| IN12 | TESTED | 307 | image_sha256 | test_api.c:896 (zero); nightly python/tests/test_api.py:61. **Finding**: the header does not say it is all zero (no image) |
361361
| IN13 | TESTED | 308 | name "NUL-terminated" | test_e2e.c:207 (the file's name), :270 (a directory's name) |
@@ -428,7 +428,7 @@ Definition (not counted): 375 TOKS_PAR_HAS_INFO.
428428
named: unigram.c:381); the header now says so (this PR: "unigram pieces: 127, -3 above"; L2c).
429429
- toks.h:4: in 0.x additive changes bumped TOKS_ABI_MINOR; the sentence names only TOKS_ABI_MAJOR (V1).
430430
- toks.h:50: TOKS_E_FORMAT's comment still names the .toks image and its checksum (E3).
431-
- toks.h:304, 307: control_isolation is always 0 and image_sha256 always zero in 0.3; the header does not say so.
431+
- toks.h:309: image_sha256 is always zero in 0.3; the header does not say so.
432432
- toks.h:133-136: mode 3 is TOKS_E_ARG (test_api.c:488, test_split.c:258, test_par.c:355); the header lists three
433433
modes and says nothing of the fourth value.
434434
- toks.h:330: toks_get_info's return (0; TOKS_E_ARG for NULL or a wrong size) is not written in the header; the size

‎include/toks.h‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -301,7 +301,9 @@ typedef struct toks_info {
301301
uint32_t paths; /* bitset of TOKS_PATH_*: stages running on compiled fast paths */
302302
uint64_t cpu_features; /* the platform layer's feature bits */
303303
uint64_t max_text; /* TOKS_MAX_TEXT */
304-
uint32_t control_isolation; /* 1 when SPEC §3.6 is certified for this tokenizer */
304+
uint32_t control_isolation; /* 1 when SPEC §3.6 is certified for this tokenizer: no added token's id comes
305+
out of a TOKS_ADDED_NONE encode of any bytes (proven at load, byte-level bpe;
306+
0: not proven, docs/algorithms/reach.md) */
305307
uint32_t rsv;
306308
uint8_t source_sha256[32];
307309
uint8_t image_sha256[32];

‎src/core/api.c‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -809,7 +809,7 @@ int64_t toks_get_info(const toks_ctx *ctx, toks_info *o)
809809
i.paths = (ctx->gen != NULL ? 0u : TOKS_PATH_SCAN) | TOKS_PATH_NORMALIZE; /* the generic engine is no fast path */
810810
i.cpu_features = ctx->cpu_features; /* read at load: no syscall here (SPEC §4.1) */
811811
i.max_text = TOKS_MAX_TEXT;
812-
i.control_isolation = 0u; /* not certified yet (SPEC §3.6) */
812+
i.control_isolation = ctx->ctrl_iso; /* SPEC §3.6, proven at load (reach.c) */
813813
memcpy(i.source_sha256, ctx->source_sha256, 32u); /* image_sha256 stays 0: no .toks image */
814814
memcpy(i.name, ctx->name, sizeof i.name);
815815
i.name[sizeof i.name - 1u] = 0;

‎src/core/config.c‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -954,6 +954,17 @@ static int64_t bl_pretok(const jv *root, toks_config *cfg, toks_arena *ar, toks_
954954
r = toks_fail(err, TOKS_E_UNSUPPORTED, n == 0u ? "pre_tokenizer absent (byte-level bpe needs ByteLevel)" : "pre_tokenizer without ByteLevel (byte-level bpe)");
955955
}
956956
if (r != 0) { return r; }
957+
cfg->reach_kind = TOKS_REACH_WHOLE; /* SPEC §3.6's last splitter (reach.c) */
958+
for (uint32_t i = 0, dom = 0u; i < n; i++) { /* bound: 16 steps */
959+
if (g[i].kind == TOKS_GS_BYTELEVEL && !g[i].lit) { dom = 1u; continue; }
960+
cfg->reach_kind = g[i].kind == TOKS_GS_BYTELEVEL ? TOKS_REACH_GPT2 :
961+
(g[i].kind == TOKS_GS_SPLIT && g[i].beh == TOKS_GB_ISOLATED && !g[i].inv) ? TOKS_REACH_SPLIT : TOKS_REACH_NONE;
962+
cfg->reach_s = g[i].s;
963+
cfg->reach_len = g[i].n;
964+
cfg->reach_lit = g[i].lit;
965+
cfg->reach_bytes = dom;
966+
dom |= (uint32_t)(g[i].kind == TOKS_GS_BYTELEVEL);
967+
}
957968
/* the compiled templates first (kernels.md §3): gpt-2, digits-gpt2, one known Split (Isolated, or Removed +
958969
* invert: docs/breadth.md §1.1), MiniCPM5's two, dsv3's three; else the generic engine */
959970
const toks_gen_spec *z = &g[n - 1u];

‎src/core/config.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@ extern const toks_pattern TOKS_PATTERN_P20;
3131
extern const char *const TOKS_DSV3_SPLITS[3];
3232
extern const toks_pattern TOKS_PATTERN_DSV3;
3333

34+
enum { TOKS_REACH_NONE, TOKS_REACH_WHOLE, TOKS_REACH_SPLIT, TOKS_REACH_GPT2 };
35+
3436
/* --- added tokens --------------------------------------------------------------------------------------- */
3537

3638
/* rationale: docs/notes/c-core.md §config.h.3 */
@@ -102,6 +104,13 @@ typedef struct toks_config {
102104
const struct toks_gen *gen; /* else, a chain no template compiles: gen.c's program (parse memory), its size */
103105
uint64_t gen_bytes;
104106

107+
/* SPEC §3.6 (reach.c, docs/algorithms/reach.md): the chain's last splitting step, which makes every final piece:
108+
* TOKS_REACH_NONE (no proof about pieces), _WHOLE (no step splits: every text is a piece), _SPLIT (an Isolated,
109+
* uninverted Split: reach_s[0, reach_len) its Regex, or its String when reach_lit; reach_bytes: after a ByteLevel,
110+
* in the byte domain), _GPT2 (ByteLevel use_regex: gpt-2's regex, Isolated) */
111+
const uint8_t *reach_s;
112+
uint32_t reach_len, reach_lit, reach_bytes, reach_kind;
113+
105114
/* rationale: docs/notes/c-core.md §config.h.5 */
106115
uint32_t cut_chunk;
107116
uint32_t cut_run;

‎src/core/core.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,7 @@ struct toks_ctx {
320320
uint32_t n_pp_prefix; /* ids before $A */
321321
uint32_t n_pp_suffix; /* ids after $A (n_pp_prefix + n_pp_suffix <= 64) */
322322
uint32_t n_nonspecial; /* added-token entries without TOKS_AF_SPECIAL (0: NONSPECIAL == NONE) */
323+
uint32_t ctrl_iso; /* SPEC §3.6 certified at load (reach.c): toks_info.control_isolation */
323324
uint32_t dec_max; /* the longest toks_decode of one id (load.c, toks_dec_max): the weight of
324325
toks_stream_bound (stream.c) */
325326
uint64_t identity; /* the scratch binding (toks_ctx_identity) */
@@ -679,6 +680,8 @@ struct toks_gen;
679680
int64_t toks_gen_compile(const toks_gen_spec *sp, uint32_t n, toks_arena *ar, const struct toks_gen **out,
680681
uint64_t *bytes, toks_err *err);
681682
uint64_t toks_gen_scr(const struct toks_gen *g);
683+
int toks_gen_piece(const struct toks_gen *g, const uint8_t *b, uint64_t n); /* reach.c's piece test (SPEC §3.6) */
684+
uint32_t toks_reach_certify(const struct toks_config *cfg); /* reach.c: SPEC §3.6 for this file, 1 = certified */
682685
void toks_gen_run(const struct toks_ctx *ctx, toks_scratch *h, const uint8_t *seg, uint64_t len, uint64_t base,
683686
toks_emit *e, int ids);
684687
/* one round of consecutive pieces seg[pos, ends[0]), [ends[0], ends[1]), .. (n >= 1): their ends (base + end) or

‎src/core/gen.c‎

Lines changed: 74 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -437,7 +437,7 @@ typedef struct gvm {
437437
} gvm;
438438

439439
/* the byte-level alphabet's char of byte b (gpt-2's bytes_to_unicode) */
440-
static uint32_t byte_char(uint32_t b) { return b <= 0x20u ? 256u + b : (b >= 0x7Fu && b <= 0xA0u) ? b + 210u : b == 0xADu ? 323u : b; }
440+
static uint32_t byte_char(uint32_t b) { return b <= 0x20u ? 256u + b : (b >= 0x7Fu && b <= 0xA0u) ? b + 162u : b == 0xADu ? 323u : b; }
441441

442442
/* the code point at p (*k bytes): 0x110000 + byte for an ill-formed byte; in the byte domain a byte's char */
443443
static uint32_t atom(const gvm *v, uint64_t p, uint32_t *k)
@@ -458,18 +458,20 @@ static uint32_t atom_before(const gvm *v, uint64_t p)
458458
}
459459

460460
/* class k holds code point c; an ill-formed byte is \p{P} (SPEC §3.3: class P) and no literal */
461-
static int has(const gvm *v, const gcls *k, uint32_t c)
461+
static int has_in(const uint32_t *rng, const gcls *k, uint32_t c)
462462
{
463463
if (c < 128u) { return (int)((k->ascii[c >> 5] >> (c & 31u)) & 1u); }
464464
uint32_t in = (c < 0x110000u ? toks_ucd_flags(c) : TOKS_UCD_P) & k->mask, lo = 0u, hi = c < 0x110000u ? k->nr : 0u;
465-
const uint32_t *r = v->rng + 2u * k->r0;
465+
const uint32_t *r = rng + 2u * k->r0;
466466
while (in == 0u && lo < hi) { /* bound: log2 nr */
467467
uint32_t m = (lo + hi) >> 1;
468468
if (r[2u * m + 1u] < c) { lo = m + 1u; } else if (r[2u * m] > c) { hi = m; } else { in = 1u; }
469469
}
470470
return (int)((in != 0u) ^ k->neg);
471471
}
472472

473+
static int has(const gvm *v, const gcls *k, uint32_t c) { return has_in(v->rng, k, c); }
474+
473475
static int word(uint32_t c) { return c < 0x110000u && (toks_ucd_flags(c) & TOKS_UCD_WORD) != 0u; }
474476

475477
static int assert_ok(const gvm *v, uint32_t kind, uint64_t p)
@@ -658,3 +660,72 @@ void toks_gen_run(const toks_ctx *ctx, toks_scratch *h, const uint8_t *seg, uint
658660
piece(&v, 0u, 0u, len);
659661
flush(&v);
660662
}
663+
664+
/* ---- reachability (SPEC §3.6, docs/algorithms/reach.md): can a string be a piece of a step? ------------------- */
665+
666+
/* the code p[0, n) of one step reaches from pc without a character, into set (a mark per pc); relaxed: an assertion
667+
* or a lookahead may pass (the text around a piece is anyone's), else neither does */
668+
static void reach_close(const gins *p, uint32_t n, uint8_t *set, uint32_t *stk, uint32_t pc, int relaxed)
669+
{
670+
uint32_t sp = 0u;
671+
stk[sp++] = pc;
672+
while (sp > 0u) { /* bound: 2 n pushes (each pc once) */
673+
pc = stk[--sp];
674+
if (pc >= n || set[pc]) { continue; }
675+
const gins *in = &p[pc];
676+
set[pc] = 1u;
677+
if (in->op == G_SPLIT) { stk[sp++] = (uint32_t)((int32_t)pc + in->y); }
678+
if (in->op == G_SPLIT || in->op == G_JMP) { stk[sp++] = (uint32_t)((int32_t)pc + in->x); }
679+
if (relaxed && (in->op == G_ASSERT || in->op == G_LOOK)) { stk[sp++] = pc + 1u + (in->op == G_LOOK ? (uint32_t)in->x : 0u); }
680+
}
681+
}
682+
683+
/* 1 unless b[0, n) is proven never to be a piece of program g's one step (an Isolated, uninverted regex split; any
684+
* other program: 1): a piece is a match or a gap between matches (generic.md §5). A match of exactly b's atoms needs
685+
* a path of the code through them with the assertions and lookaheads relaxed (none: no match); a gap holds no position
686+
* where a match starts, so an ascii atom (in the byte domain: any byte, whose char needs no context) from which the
687+
* code reaches its G_MATCH through that one char and no assertion makes b no gap. */
688+
int toks_gen_piece(const struct toks_gen *g, const uint8_t *b, uint64_t n)
689+
{
690+
const gstep *st = &g->step[0];
691+
if (g->n_steps != 1u || st->kind != GS_RX || st->beh != TOKS_GB_ISOLATED || st->inv || n == 0u) { return 1; }
692+
const gins *p = G_INS(g) + st->pc0;
693+
const gcls *cls = G_CLS(g);
694+
const uint32_t *rng = G_RNG(g);
695+
uint32_t L = st->len, stk[2u * G_MAX_STEP + 2u];
696+
uint8_t cur[G_MAX_STEP], nxt[G_MAX_STEP], one[G_MAX_STEP];
697+
int live = 1, gap = 1;
698+
memset(cur, 0, L);
699+
reach_close(p, L, cur, stk, 0u, 1);
700+
for (uint64_t i = 0u; i < n && live;) { /* bound: n (each atom >= 1 byte) */
701+
uint32_t k = st->bytes ? 1u : toks_utf8_len(b + i, n - i), c; /* the VM's atom (atom above) */
702+
c = st->bytes ? byte_char(b[i]) : k == 0u ? 0x110000u + b[i] : k == 1u ? b[i] : toks_cp_decode(b + i, k);
703+
memset(nxt, 0, L);
704+
live = 0;
705+
for (uint32_t pc = 0u; pc < L; pc++) { /* bound: L */
706+
if (cur[pc] && p[pc].op == G_CHAR && has_in(rng, &cls[p[pc].x], c)) { reach_close(p, L, nxt, stk, pc + 1u, 1); live = 1; }
707+
}
708+
memcpy(cur, nxt, L);
709+
i += (k != 0u) ? k : 1u;
710+
}
711+
for (uint32_t pc = 0u; live && pc < L; pc++) { /* bound: L */
712+
if (cur[pc] && p[pc].op == G_MATCH) { return 1; } /* b can be a match */
713+
}
714+
memset(cur, 0, L); /* the gap: the strict start closure, and the */
715+
reach_close(p, L, cur, stk, 0u, 0); /* chars after which G_MATCH follows strictly */
716+
for (uint32_t pc = 0u; pc < L; pc++) { /* bound: L */
717+
one[pc] = 0u;
718+
if (!cur[pc] || p[pc].op != G_CHAR) { continue; }
719+
memset(nxt, 0, L);
720+
reach_close(p, L, nxt, stk, pc + 1u, 0);
721+
for (uint32_t q = 0u; q < L; q++) { if (nxt[q] && p[q].op == G_MATCH) { one[pc] = 1u; } } /* bound: L */
722+
}
723+
for (uint64_t i = 0u; gap && i < n; i++) { /* bound: n */
724+
if (!st->bytes && b[i] >= 0x80u) { continue; } /* a multi-byte atom's char needs its context */
725+
uint32_t c = st->bytes ? byte_char(b[i]) : b[i];
726+
for (uint32_t pc = 0u; gap && pc < L; pc++) { /* bound: L */
727+
if (one[pc] && has_in(rng, &cls[p[pc].x], c)) { gap = 0; } /* a match starts at this atom */
728+
}
729+
}
730+
return gap;
731+
}

0 commit comments

Comments
 (0)