tools: decode cancelled-extended terminfo layout -- infocmp/tic -x now 100%

Crack the extended (user-defined) offset-table layout that previously bounded the
`-x` paths, lifting both `infocmp -1 -x` and the `infocmp -1 -x | tic -x` round-trip
to 100.000% byte-exact across the whole 2,869-entry terminfo database.

The key was the offset-table length: it is `2*es + eb + en` (es string-value
offsets followed by eb+en+es cap-name offsets), NOT the extended header's 4th field
-- which is the *count of strings* in the table (offsets minus the cancelled string
caps, since a cancelled string keeps a -2 value slot but contributes no value
string). The 5th field is the string-table byte size. Byte accounting
(`p + 2*(2es+eb+en) + str_size == file_end`) confirms it.

reader (src/terminfo/mod.rs): read `2*es+eb+en` offsets; surface extended string
caps with cancellation (`Some(value)` vs `None` for `name@`); `ext_string_caps()`.

infocmp: emit cancelled extended strings as `name@`, merged sorted.

tic: parse extended caps (names outside the predefined tables; a bare `name@` is a
cancelled extended string, matching ncurses); with `-x`, lift the SVr4 predefined
cutoff and append the full extended section -- ext bool bytes (+pad), ext numbers,
the value+name offset tables, and the value/name string tables, with the
str_count/str_size header and cancelled `-2` value offsets.

Both NCURSES.INFOCMP and NCURSES.TIC courts now admit 100.000% on -1 and -x.
Closes the cancelled-extended-capability corner noted in the gap ledger; advances
BLD-03.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbiPgWC3PHKmrTNvZzXcWQ

Author: claude

README.md

@@ -82,12 +82,14 @@ The same crate ships a native **`infocmp`** decompiler whose `-1` source output
 byte-exact across the whole 2,869-entry terminfo database** (`NCURSES.INFOCMP` court): it reconstructs
 `_nc_tic_expand` string escaping (the caret-vs-octal length heuristic, the `%`-operator verbatim rule,
 the `\E`/`\n`/`\r`/`\0`/`\s`/`\,`/`\^` escapes), power-of-two hex number formatting (`colors#0x100`),
-cancelled caps (`name@`), and `acsc` glyph-pair sorting. `infocmp -1 -x` (extended caps) is at 99.2%,
-the residual confined to entries using *cancelled extended* capabilities (see the gap ledger). Its
-inverse, a native **`tic`**, compiles source back to the binary form **100.000% byte-identically to
-system tic across the whole database** on the `infocmp -1 | tic` round-trip (`NCURSES.TIC` court):
-the source parser, the `_nc_tic_expand` un-escaper, and the binary writer (magic selection, the SVr4
-cutoff that drops ncurses-extension predefined caps without `-x`, cancelled `-2` markers).
+cancelled caps (`name@`), `acsc` glyph-pair sorting, and the full extended (user-defined, `-x`)
+section including cancelled `name@` extensions — **100.000% byte-exact for both `-1` and `-1 -x`
+across the whole database** (`NCURSES.INFOCMP` court). Its inverse, a native **`tic`**, compiles
+source back to the binary form **100.000% byte-identically to system tic** on both the
+`infocmp -1 | tic` and `infocmp -1 -x | tic -x` round-trips (`NCURSES.TIC` court): the source parser,
+the `_nc_tic_expand` un-escaper, and the binary writer (magic selection, the SVr4 cutoff that drops
+ncurses-extension predefined caps without `-x`, cancelled `-2` markers, and the extended-section
+offset/string tables).
 
 ## Forensic gap ledger
 

crates/ncurses-tools/src/bin/infocmp.rs

@@ -277,10 +277,14 @@ fn main() {
     }
 
     if ext {
-        let mut ex: Vec<(String, Vec<u8>)> = ti.ext_string_names();
+        // Extended string caps follow, sorted by name; a cancelled one renders as `name@`.
+        let mut ex = ti.ext_string_caps();
         ex.sort_unstable_by(|a, b| a.0.cmp(&b.0));
         for (n, v) in ex {
-            out.push_str(&format!("\t{}={},\n", n, tic_expand(&v)));
+            match v {
+                Some(bytes) => out.push_str(&format!("\t{}={},\n", n, tic_expand(&bytes))),
+                None => out.push_str(&format!("\t{n}@,\n")),
+            }
         }
     }
 

crates/ncurses-tools/src/bin/tic.rs

@@ -86,7 +86,7 @@ fn unescape(s: &[u8]) -> Vec<u8> {
     out
 }
 
-/// One parsed capability from source.
+/// One parsed *predefined* capability from source.
 enum Cap {
     Bool(usize),
     Num(usize, i32),
@@ -95,10 +95,20 @@ enum Cap {
     Cancel { is_string: bool, index: usize },
 }
 
+/// An *extended* (user-defined) capability -- a name not in the predefined tables. A bare `name@`
+/// is, like ncurses, a cancelled extended *string*.
+enum ExtCap {
+    Bool(String),
+    Num(String, i32),
+    /// `Some(bytes)` present value, `None` cancelled (`name@`).
+    Str(String, Option<Vec<u8>>),
+}
+
 /// A parsed terminfo entry ready to compile.
 struct Entry {
     names: String,
     caps: Vec<Cap>,
+    ext: Vec<ExtCap>,
 }
 
 /// Split an entry body into comma-separated fields. A `\X` escape and a `^X` caret pair are each
@@ -165,32 +175,43 @@ fn parse_entry(text: &str) -> Option<Entry> {
         return None;
     }
     let mut caps_out = Vec::new();
+    let mut ext_out = Vec::new();
     for f in it {
         let f = f.trim();
         if f.is_empty() || f.starts_with("use=") {
             continue; // `use=` flattening is not needed for resolved (`infocmp -1`) source.
         }
         if let Some(eq) = f.find('=') {
             let (name, val) = (&f[..eq], &f[eq + 1..]);
-            if let Some(ix) = str_index(name) {
-                caps_out.push(Cap::Str(ix, unescape(val.as_bytes())));
+            match str_index(name) {
+                Some(ix) => caps_out.push(Cap::Str(ix, unescape(val.as_bytes()))),
+                None => ext_out.push(ExtCap::Str(name.to_string(), Some(unescape(val.as_bytes())))),
             }
         } else if let Some(name) = f.strip_suffix('@') {
             if let Some(ix) = str_index(name) {
                 caps_out.push(Cap::Cancel { is_string: true, index: ix });
             } else if let Some(ix) = num_index(name) {
                 caps_out.push(Cap::Cancel { is_string: false, index: ix });
+            } else {
+                // An unknown cancelled cap is, like ncurses, a cancelled extended string.
+                ext_out.push(ExtCap::Str(name.to_string(), None));
             }
         } else if let Some(h) = f.find('#') {
             let (name, val) = (&f[..h], &f[h + 1..]);
-            if let (Some(ix), Some(n)) = (num_index(name), parse_num(val)) {
-                caps_out.push(Cap::Num(ix, n));
+            if let Some(n) = parse_num(val) {
+                match num_index(name) {
+                    Some(ix) => caps_out.push(Cap::Num(ix, n)),
+                    None => ext_out.push(ExtCap::Num(name.to_string(), n)),
+                }
+            }
+        } else {
+            match bool_index(f) {
+                Some(ix) => caps_out.push(Cap::Bool(ix)),
+                None => ext_out.push(ExtCap::Bool(f.to_string())),
             }
-        } else if let Some(ix) = bool_index(f) {
-            caps_out.push(Cap::Bool(ix));
         }
     }
-    Some(Entry { names, caps: caps_out })
+    Some(Entry { names, caps: caps_out, ext: ext_out })
 }
 
 /// SVr4 capability counts: without `-x`, `tic` stores only caps below these indices. The
@@ -201,7 +222,15 @@ const SVR4_NUM: usize = 33;
 const SVR4_STR: usize = 394;
 
 /// Compile a parsed entry to the binary terminfo layout (the inverse of this crate's reader).
-fn compile(entry: &Entry) -> Vec<u8> {
+/// With `ext`, the predefined-cap cutoff is lifted (the full tables are stored) and an extended
+/// (user-defined) section is appended; without it, the SVr4 cutoff applies and extensions drop.
+fn compile(entry: &Entry, ext: bool) -> Vec<u8> {
+    let (cut_b, cut_n, cut_s) = if ext {
+        (caps::BOOL_NAMES.len(), caps::NUM_NAMES.len(), caps::STR_NAMES.len())
+    } else {
+        (SVR4_BOOL, SVR4_NUM, SVR4_STR)
+    };
+
     // Lay capabilities out by index; absent slots are -1, cancelled are -2.
     let mut bool_vals: Vec<i8> = Vec::new(); // 0 absent, 1 true
     let mut num_vals: Vec<i32> = Vec::new(); // -1 absent, -2 cancelled, else value
@@ -227,27 +256,27 @@ fn compile(entry: &Entry) -> Vec<u8> {
 
     for cap in &entry.caps {
         match cap {
-            Cap::Bool(ix) if *ix < SVR4_BOOL => {
+            Cap::Bool(ix) if *ix < cut_b => {
                 ensure_b(&mut bool_vals, *ix);
                 bool_vals[*ix] = 1;
             }
-            Cap::Num(ix, n) if *ix < SVR4_NUM => {
+            Cap::Num(ix, n) if *ix < cut_n => {
                 ensure_n(&mut num_vals, *ix);
                 num_vals[*ix] = *n;
             }
-            Cap::Str(ix, bytes) if *ix < SVR4_STR => {
+            Cap::Str(ix, bytes) if *ix < cut_s => {
                 ensure_s(&mut str_vals, &mut str_cancel, *ix);
                 str_vals[*ix] = Some(bytes.clone());
             }
-            Cap::Cancel { is_string: true, index } if *index < SVR4_STR => {
+            Cap::Cancel { is_string: true, index } if *index < cut_s => {
                 ensure_s(&mut str_vals, &mut str_cancel, *index);
                 str_cancel[*index] = true;
             }
-            Cap::Cancel { is_string: false, index } if *index < SVR4_NUM => {
+            Cap::Cancel { is_string: false, index } if *index < cut_n => {
                 ensure_n(&mut num_vals, *index);
                 num_vals[*index] = -2;
             }
-            // Caps at/above the SVr4 cutoffs are ncurses extensions, stored only under `-x`.
+            // Caps at/above the cutoff are ncurses extensions, stored only under `-x`.
             _ => {}
         }
     }
@@ -256,8 +285,13 @@ fn compile(entry: &Entry) -> Vec<u8> {
     let nnum = num_vals.len();
     let nstr = str_vals.len();
 
-    // 32-bit numbers only when a value does not fit a signed 16-bit (ncurses' magic switch).
-    let need_32 = num_vals.iter().any(|&n| n > 32767);
+    // 32-bit numbers only when a value does not fit a signed 16-bit (ncurses' magic switch);
+    // extended numerics count too.
+    let ext_num_max = entry.ext.iter().filter_map(|e| match e {
+        ExtCap::Num(_, n) => Some(*n),
+        _ => None,
+    });
+    let need_32 = num_vals.iter().copied().chain(ext_num_max).any(|n| n > 32767);
     let (magic, num_width) = if need_32 {
         (MAGIC_EXTENDED_NUMBERS, 4usize)
     } else {
@@ -313,9 +347,102 @@ fn compile(entry: &Entry) -> Vec<u8> {
         out.extend_from_slice(&o.to_le_bytes());
     }
     out.extend_from_slice(&strtab);
+
+    if ext && !entry.ext.is_empty() {
+        append_extended(&mut out, &entry.ext, num_width);
+    }
     out
 }
 
+/// Append the extended (user-defined) section, mirroring ncurses' `-x` layout: an even-aligned
+/// header `[eb, en, es, str_count, str_size]`, the extended boolean bytes (+pad), the extended
+/// numbers, an offset table of `es` value offsets then `eb+en+es` name offsets, and a string table
+/// of the present value strings followed by every cap name. Cancelled strings keep a `-2` value
+/// offset and no value string.
+fn append_extended(out: &mut Vec<u8>, ext: &[ExtCap], num_width: usize) {
+    let bools: Vec<&String> = ext
+        .iter()
+        .filter_map(|e| if let ExtCap::Bool(n) = e { Some(n) } else { None })
+        .collect();
+    let nums: Vec<(&String, i32)> = ext
+        .iter()
+        .filter_map(|e| if let ExtCap::Num(n, v) = e { Some((n, *v)) } else { None })
+        .collect();
+    let strs: Vec<(&String, &Option<Vec<u8>>)> = ext
+        .iter()
+        .filter_map(|e| if let ExtCap::Str(n, v) = e { Some((n, v)) } else { None })
+        .collect();
+    let (eb, en, es) = (bools.len(), nums.len(), strs.len());
+
+    // Value strings (present only) then names (every cap). Offsets: values from the table start,
+    // names relative to the end of the value region.
+    let mut value_tab: Vec<u8> = Vec::new();
+    let mut val_offsets: Vec<i16> = Vec::with_capacity(es);
+    for (_, v) in &strs {
+        match v {
+            Some(bytes) => {
+                val_offsets.push(value_tab.len() as i16);
+                value_tab.extend_from_slice(bytes);
+                value_tab.push(0);
+            }
+            None => val_offsets.push(-2),
+        }
+    }
+    let mut name_tab: Vec<u8> = Vec::new();
+    let mut name_offsets: Vec<i16> = Vec::with_capacity(eb + en + es);
+    let push_name = |tab: &mut Vec<u8>, offs: &mut Vec<i16>, name: &str| {
+        offs.push(tab.len() as i16);
+        tab.extend_from_slice(name.as_bytes());
+        tab.push(0);
+    };
+    for n in &bools {
+        push_name(&mut name_tab, &mut name_offsets, n);
+    }
+    for (n, _) in &nums {
+        push_name(&mut name_tab, &mut name_offsets, n);
+    }
+    for (n, _) in &strs {
+        push_name(&mut name_tab, &mut name_offsets, n);
+    }
+
+    let present_values = val_offsets.iter().filter(|&&o| o >= 0).count();
+    let str_count = present_values + eb + en + es;
+    let str_size = value_tab.len() + name_tab.len();
+
+    // The extended section starts on an even boundary.
+    if out.len() % 2 == 1 {
+        out.push(0);
+    }
+    let push16 = |o: &mut Vec<u8>, v: u16| o.extend_from_slice(&v.to_le_bytes());
+    push16(out, eb as u16);
+    push16(out, en as u16);
+    push16(out, es as u16);
+    push16(out, str_count as u16);
+    push16(out, str_size as u16);
+    for n in &bools {
+        let _ = n;
+        out.push(1); // extended booleans present in source are true
+    }
+    if eb % 2 == 1 {
+        out.push(0);
+    }
+    for (_, v) in &nums {
+        if num_width == 2 {
+            out.extend_from_slice(&(*v as i16).to_le_bytes());
+        } else {
+            out.extend_from_slice(&v.to_le_bytes());
+        }
+    }
+    for &o in &val_offsets {
+        out.extend_from_slice(&o.to_le_bytes());
+    }
+    for &o in &name_offsets {
+        out.extend_from_slice(&o.to_le_bytes());
+    }
+    out.extend_from_slice(&value_tab);
+    out.extend_from_slice(&name_tab);
+}
+
 fn split_entries(text: &str) -> Vec<String> {
     // Strip comment lines, then group: an entry begins at a non-indented, non-empty line.
     let mut entries = Vec::new();
@@ -341,6 +468,7 @@ fn main() {
     let argv: Vec<String> = std::env::args().skip(1).collect();
     let mut out_dir: Option<String> = None;
     let mut input: Option<String> = None;
+    let mut ext = false;
     let mut i = 0;
     while i < argv.len() {
         match argv[i].as_str() {
@@ -349,8 +477,9 @@ fn main() {
                 i += 1;
             }
             a if a.starts_with("-o") && a.len() > 2 => out_dir = Some(a[2..].to_string()),
+            "-x" => ext = true,
             // Accept and ignore the common flags that do not change compiled bytes.
-            "-x" | "-1" | "-v" | "-s" | "-c" | "-r" | "-a" | "-g" | "-q" => {}
+            "-1" | "-v" | "-s" | "-c" | "-r" | "-a" | "-g" | "-q" => {}
             a if a.starts_with('-') && a != "-" => {}
             a => input = Some(a.to_string()),
         }
@@ -387,7 +516,7 @@ fn main() {
         if primary.is_empty() {
             continue;
         }
-        let compiled = compile(&entry);
+        let compiled = compile(&entry, ext);
         let sub = &primary[..1];
         let subdir = PathBuf::from(&dir).join(sub);
         if let Err(e) = std::fs::create_dir_all(&subdir) {

docs/gap-ledger.md

@@ -588,24 +588,25 @@ intentional, documented divergence — that is strictly-better parity, not a gap
   `tput`, `clear`, `tabs`, `reset`, `tset`, `captoinfo`, `infotocap` — scripts
   depend on `tput`/`clear` exit codes & output, and on `infocmp | tic` round-trip
   idempotence. **`tput`/`clear` are done** (native binaries, `NCURSES.TPUT` court).
-  **`infocmp` is done for the `-1` source layout: 100.000% byte-exact across the
+  **`infocmp` is done — 100.000% byte-exact for BOTH `-1` and `-1 -x` across the
   whole 2,869-entry terminfo database** (`NCURSES.INFOCMP` court) — the decompiler
   reconstructs `_nc_tic_expand` escaping (the caret-vs-octal length heuristic, the
   `%`-operator verbatim rule, the letter/`\s`/`\,`/`\^` escapes), power-of-two hex
-  number formatting (`colors#0x100`), cancelled caps (`name@`), and `acsc` glyph
-  sorting. `infocmp -1 -x` (extended/user-defined caps) is at **99.2%** (2846/2869);
-  the residual is confined to entries that use *cancelled extended* capabilities,
-  whose compiled offset-table layout is not yet decoded byte-exactly (honest bound,
-  not a claimed match). **`tic` is now done for the non-extended case: 100.000%
-  byte-exact across the whole 2,869-entry database on the `infocmp -1 | tic`
-  round-trip** (`NCURSES.TIC` court) — the inverse of `infocmp`, reconstructing the
+  number formatting (`colors#0x100`), cancelled caps (`name@`), `acsc` glyph
+  sorting, and the full extended (user-defined, `-x`) section including cancelled
+  `name@` extensions — the extended offset-table layout (value offsets + name
+  offsets = `2·es + eb + en`, the header's string-*count* vs string-*size* fields,
+  cancelled strings keeping a `-2` value slot but no value string) is now decoded
+  byte-exactly. **`tic` is done too — 100.000% byte-exact on BOTH the
+  `infocmp -1 | tic` and `infocmp -1 -x | tic -x` round-trips across the whole
+  database** (`NCURSES.TIC` court) — the inverse of `infocmp`, reconstructing the
   source parser (comma/`^X`/`\X`-aware splitting), the `_nc_tic_expand` un-escaper
   (`\E`/`^X`/`\nnn`/`\0`→0x80/`\s`, the `%`-operator verbatim rule), and the binary
   writer (header, `|`-names, bool bytes, even-offset numbers with 16-vs-32-bit magic
-  selection, string offsets/table, cancelled `-2`, and the SVr4 cutoff that drops
-  ncurses-extension predefined caps without `-x`). `tic -x` (extended-cap
-  compilation) shares the same cancelled-extended-offset-table corner as `infocmp -x`.
-  `toe`/`tabs`/`reset`/`tset`/`captoinfo`/`infotocap` remain. **S1, in progress.**
+  selection, string offsets/table, cancelled `-2`, the SVr4 cutoff that drops
+  ncurses-extension predefined caps without `-x`, and — with `-x` — the full
+  extended section writer). `toe`/`tabs`/`reset`/`tset`/`captoinfo`/`infotocap`
+  remain. **S1, in progress.**
 - **BLD-04 · companion libraries panel / menu / form (+`w`)** — ~200+ symbols,
   entire subsystems (overlapping-window stack; menu driver; field/form driver).
   **Zero coverage.** **S0, open.**
@@ -1160,7 +1161,7 @@ intentional, documented divergence — that is strictly-better parity, not a gap
    draws each field's buffer padded into the (sub)window and form_driver edits the current field's
    buffer + parks the cursor; byte-identical to system `libformw`, `NCURSES.FORM` court). All three
    companion libraries are built natively (closes BLD-04 / STRUCT-03).
-5. **CLI tools** — `tput`/`clear` are **done** (native binaries in `crates/ncurses-tools`, byte-identical to the system tools incl. the extended-terminfo `E3` append, `NCURSES.TPUT` court); **`infocmp -1` is done** (100.000% byte-exact decompile across the 2,869-entry terminfo DB, `NCURSES.INFOCMP` court; `-1 -x` at 99.2%, residual = cancelled extended caps); **`tic` is done** for the non-extended case (100.000% byte-exact `infocmp -1 | tic` round-trip across the DB, `NCURSES.TIC` court); `toe`/… remain (BLD-03).
+5. **CLI tools** — `tput`/`clear` are **done** (native binaries in `crates/ncurses-tools`, byte-identical to the system tools incl. the extended-terminfo `E3` append, `NCURSES.TPUT` court); **`infocmp` is done** (100.000% byte-exact decompile for both `-1` and `-1 -x` across the 2,869-entry terminfo DB, `NCURSES.INFOCMP` court); **`tic` is done** (100.000% byte-exact on both the `infocmp -1 | tic` and `infocmp -1 -x | tic -x` round-trips, `NCURSES.TIC` court); `toe`/… remain (BLD-03).
 6. **Native API completeness** — the macro API as Rust methods, panel/menu/form
    modules, wide-char module, and the CLI tools as native binaries (closes
    STRUCT-03); plus a global-state compatibility shim (thread-local `SP`/`cur_term`,