feat(html): key-value form regions (docling field_region/field_item)

MIGRATION.md

@@ -66,7 +66,7 @@ PyPI; run via `scripts/conformance.sh <fmt>`), not the committed groundtruth
 |---|---|---|
 | Markdown | `markdown.rs` (pulldown-cmark) | **10/10 exact** |
 | CSV | `csv.rs` (`csv` crate) | **9/9 exact** |
-| HTML | `html.rs` (scraper/html5ever) | **28/32 exact** (rest need a headless browser — §5) |
+| HTML | `html.rs` (scraper/html5ever) | **30/32 exact** (rest need a headless browser — §5) |
 | AsciiDoc | `asciidoc.rs` (regex) | **4/4 exact** |
 | DeepSeek-OCR Markdown | `deepseek.rs` | **3/3 exact** (auto-detected VLM-token variant) |
 | XLSX | `xlsx.rs` (calamine) | **9/9 exact** |
@@ -187,9 +187,11 @@ These are deliberate or unavoidable divergences, not bugs.
    the converter routes by content markers (`us-patent` → USPTO, `us-gaap`/`dei`
    → XBRL, else JATS) rather than the extension alone.
 
-8. **No headless-browser pass.** A few HTML behaviours depend on rendering the
-   page (nav/visibility suppression, form key-value regions, rendered bounding
-   boxes) — see §5.
+8. **Headless-browser pass is opt-in.** Form key-value regions and inline
+   visibility are handled statically by default. Stylesheet-driven (CSS-cascade)
+   nav/visibility suppression needs a rendered page, available behind the
+   optional `web-browser` feature / `--use-web-browser` flag (Rust-driven
+   Chromium); rendered-bounding-box nested-table padding is still out — see §5.
 
 ---
 
@@ -203,9 +205,6 @@ Explicitly **not done**, with the reason:
   classification, formula understanding, code understanding). Model-bound; out of
   scope for the discriminative port.
 
-(**TableFormer is now done** — ported to ONNX and run on every table region; see
-§5 and `PDF_CONFORMANCE.md`. Geometric reconstruction remains only as the fallback
-when the TableFormer graphs aren't present.)
 - **XML DocLang / DocTags** input backend — no `.dclg` sources in the corpus to
   verify against, and not in the requested scope.
 - **Older patent schemas.** USPTO covers the modern `v4x` XML only; the
@@ -228,9 +227,22 @@ when the TableFormer graphs aren't present.)
   inline-group spacing and stay attached to their list item; `\operatorname`
   functions, limit-label space escaping and the two-space symbol padding match
   pylatexenc byte-for-byte).
-- **HTML browser-render subsystem** — nav/visibility suppression (`wiki_duck`),
-  form key-value-pair regions (`kvp_data_example`), deep nested-table cell padding
-  from rendered bounding boxes. ~4 HTML fixtures + KVP.
+- **HTML browser-render subsystem** — the browser-free parts are **done**: form
+  key-value regions (`kvp_data_example`, detected statically from docling's
+  `keyN` / `keyN_valueM` / `keyN_marker` `id`-convention), docling-faithful
+  inline-image handling (inline images emit nothing; only block / `<a>`-wrapped
+  / `<figure>` images become pictures), and inline visibility suppression
+  (`hidden` / inline `display:none` / `visibility:hidden`).
+  For stylesheet-driven (CSS-cascade) visibility — `wiki_duck`'s collapsed
+  menus, kept distinct from the still-visible table of contents — there is now
+  an **optional headless-browser pre-render** behind the `web-browser` Cargo
+  feature / `--use-web-browser` flag: it drives the system Chromium from Rust
+  (via `headless_chrome`, no Node/Playwright), strips computed-`display:none`
+  subtrees, and hands the cleaned HTML back to the Rust backend. It resolves the
+  cascade only when the page's CSS is reachable (inline `<style>`, or external
+  stylesheets fetchable with a base host), so a saved page whose stylesheets are
+  external + offline still needs the real network. Deep nested-table cell
+  padding from rendered bounding boxes remains the last rendered-geometry gap.
 
 
 ---

README.md

@@ -36,8 +36,11 @@ with [`mail-parser`](https://crates.io/crates/mail-parser) (which conforms to
 routed through the HTML backend, with embedded images resolved from the
 archive by `Content-Location`/`cid:`. The discriminative PDF/image pipeline
 lives in `fleischwolf-pdf`: a pure-Rust PDF text parser, pdfium for page
-rasterization, and an ONNX layout/TableFormer/OCR stack. Audio/ASR is the main
-format still on the roadmap (see `MIGRATION.md`).
+rasterization, and an ONNX layout/TableFormer/OCR stack. TableFormer is ported
+to ONNX and run on every detected table region to recover its structure;
+geometric reconstruction from cell positions remains only as the fallback when
+the TableFormer graphs aren't present (see `PDF_CONFORMANCE.md`). Audio/ASR is
+the main format still on the roadmap (see `MIGRATION.md`).
 
 Output is checked against upstream Python docling — declarative formats
 byte-for-byte against live docling, the ML pipeline against a deterministic
@@ -170,6 +173,35 @@ by a wide margin, but a scanned/image-only PDF (no embedded text layer) comes
 back empty rather than erroring, so a caller can detect that and re-convert
 without the flag.
 
+### Headless-browser HTML pre-render (optional)
+
+Almost everything in the HTML backend is pure Rust, but one thing a static
+parse can't do is resolve the **CSS cascade** — whether a stylesheet- or
+class-driven rule makes an element `display:none` (e.g. a collapsed nav menu).
+The optional `--use-web-browser` flag renders the page in the system Chromium
+first, drops every element the browser computes as hidden, then feeds the
+cleaned HTML through the normal Rust backend (so all structure/table/KVP/
+formatting logic still runs in Rust — the browser only decides visibility). It
+applies to every HTML-routing input: direct HTML, plus MHTML and EPUB (which
+assemble HTML from their archives). It's driven straight from Rust over the
+DevTools protocol via
+[`headless_chrome`](https://crates.io/crates/headless_chrome) — no Node,
+Playwright, or other runtime.
+
+It's gated behind the off-by-default `web-browser` Cargo feature, so the standard
+build stays browser-dependency-free:
+
+```bash
+cargo run -p fleischwolf-cli --features web-browser -- --use-web-browser page.html
+```
+
+Chromium is located via `$FLEISCHWOLF_CHROME`/`$CHROME`, then
+`$PLAYWRIGHT_BROWSERS_PATH/chromium`, else autodetected. The page's CSS must be
+reachable for the cascade to resolve — inline `<style>` works offline, but a
+saved page that links external stylesheets needs those fetchable (with a base
+host). Without the feature, `--use-web-browser` is a clear error rather than a
+silent no-op.
+
 ## Node.js / Bun bindings
 
 Fleischwolf ships as an npm package, [**`fleischwolf`**](https://www.npmjs.com/package/fleischwolf)

crates/fleischwolf-cli/Cargo.toml

@@ -15,5 +15,11 @@ rust-version.workspace = true
 name = "fleischwolf"
 path = "src/main.rs"
 
+[features]
+# Pass through to the library's optional headless-browser HTML pre-render, so
+# `cargo run -p fleischwolf-cli --features web-browser -- --use-web-browser …`
+# activates it. Off by default (keeps the default binary browser-dependency-free).
+web-browser = ["fleischwolf/web-browser"]
+
 [dependencies]
 fleischwolf = { path = "../fleischwolf", version = "0.10.0" }

crates/fleischwolf-cli/src/main.rs

@@ -3,7 +3,7 @@
 //! A stand-in for `docling.cli.main`; the full Typer-style CLI (batch mode,
 //! pipeline options) is a later phase.
 //!
-//! Usage: fleischwolf [--strict] [--to md|json] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] <input-file>
+//! Usage: fleischwolf [--strict] [--to md|json] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--use-web-browser] <input-file>
 //!   --to md|json       output format (default: md). `json` emits docling-core's
 //!                      native DoclingDocument JSON (export_to_dict).
 //!   --images MODE      picture handling for Markdown (mirrors docling's
@@ -31,6 +31,10 @@
 //!                      fastest option, but a scanned/image-only PDF (no
 //!                      embedded text layer) yields no text — convert those
 //!                      without this flag.
+//!   --use-web-browser  pre-render HTML/MHTML/EPUB in the system Chromium (driven
+//!                      from Rust) so stylesheet-driven `display:none` elements
+//!                      (e.g. a collapsed nav menu) are dropped before parsing.
+//!                      Requires building with `--features web-browser`.
 
 use std::io::{self, Write};
 use std::path::Path;
@@ -46,6 +50,7 @@ fn main() -> ExitCode {
     let mut no_stream = false;
     let mut no_table_former = false;
     let mut no_ocr = false;
+    let mut use_web_browser = false;
     let mut bench_warm: Option<usize> = None;
     let mut path: Option<String> = None;
     let mut args = std::env::args().skip(1);
@@ -56,6 +61,7 @@ fn main() -> ExitCode {
             "--no-stream" => no_stream = true,
             "--no-table-former" => no_table_former = true,
             "--no-ocr" => no_ocr = true,
+            "--use-web-browser" => use_web_browser = true,
             "--to" => to = args.next().unwrap_or_default(),
             "--images" => images = args.next().unwrap_or_default(),
             // Hidden benchmarking aid: load the PDF/image pipeline once, then time
@@ -90,7 +96,7 @@ fn main() -> ExitCode {
     };
 
     let Some(path) = path else {
-        eprintln!("usage: fleischwolf [--strict] [--to md|json] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] <input-file>");
+        eprintln!("usage: fleischwolf [--strict] [--to md|json] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--use-web-browser] <input-file>");
         return ExitCode::from(2);
     };
 
@@ -124,7 +130,8 @@ fn main() -> ExitCode {
         .strict(strict)
         .fetch_images(fetch_images)
         .no_table_former(no_table_former)
-        .no_ocr(no_ocr);
+        .no_ocr(no_ocr)
+        .use_web_browser(use_web_browser);
 
     // Stream Markdown by default: print each chunk as the converter produces it
     // (page by page for PDF). JSON needs the whole tree, and the referenced image

crates/fleischwolf-core/src/document.rs

@@ -65,6 +65,22 @@ pub enum Node {
     },
     /// A logical grouping of child nodes (e.g. a list, a section).
     Group { label: String, children: Vec<Node> },
+    /// A form key-value region (docling's `field_region`): a set of form fields,
+    /// each pairing an optional marker, key, and value. Backends detect these
+    /// from form structure (e.g. HTML's `keyN` / `keyN_valueM` / `keyN_marker`
+    /// `id`-convention); the serializers render each item's parts as separate
+    /// labelled texts (`marker` / `field_key` / `field_value`).
+    FieldRegion { items: Vec<FieldItem> },
+}
+
+/// One entry of a [`Node::FieldRegion`]: a marker/key/value triple, any of which
+/// may be absent. Mirrors docling's `field_item` with its `marker` / `field_key`
+/// / `field_value` child texts.
+#[derive(Debug, Clone, PartialEq, Default)]
+pub struct FieldItem {
+    pub marker: Option<String>,
+    pub key: Option<String>,
+    pub value: Option<String>,
 }
 
 /// An extracted picture's raw encoded bytes plus its mimetype and pixel size —

crates/fleischwolf-core/src/json.rs

@@ -97,7 +97,7 @@ pub fn to_json(doc: &DoclingDocument) -> Value {
     let mut b = Builder::default();
     let body = b.walk_into(&doc.nodes, "#/body");
 
-    json!({
+    let mut out = json!({
         "schema_name": "DoclingDocument",
         "version": SCHEMA_VERSION,
         "name": doc.name,
@@ -127,7 +127,23 @@ pub fn to_json(doc: &DoclingDocument) -> Value {
         "key_value_items": [],
         "form_items": [],
         "pages": {},
-    })
+    });
+
+    // docling only emits `field_regions` / `field_items` when a document has
+    // form fields, and places them just before `pages`. Insert them in that slot
+    // (re-appending `pages` afterwards, since `preserve_order` keeps insertion
+    // order) so non-KVP documents' JSON is byte-identical to before.
+    if !b.field_regions.is_empty() {
+        if let Some(obj) = out.as_object_mut() {
+            let pages = obj.remove("pages");
+            obj.insert("field_regions".into(), Value::Array(b.field_regions));
+            obj.insert("field_items".into(), Value::Array(b.field_items));
+            if let Some(pages) = pages {
+                obj.insert("pages".into(), pages);
+            }
+        }
+    }
+    out
 }
 
 #[derive(Default)]
@@ -136,6 +152,8 @@ struct Builder {
     groups: Vec<Value>,
     tables: Vec<Value>,
     pictures: Vec<Value>,
+    field_regions: Vec<Value>,
+    field_items: Vec<Value>,
 }
 
 impl Builder {
@@ -165,11 +183,60 @@ impl Builder {
                 Some(self.add_picture(caption.as_deref(), image.as_ref(), parent))
             }
             Node::Group { label, children } => Some(self.add_group(label, children, parent)),
+            Node::FieldRegion { items } => Some(self.add_field_region(items, parent)),
             // Handled by `add_list` in `walk`.
             Node::ListItem { .. } => None,
         }
     }
 
+    /// A form key-value region: `field_regions/N` holds the region, each field is
+    /// a `field_items/M` whose children are its `marker` / `field_key` /
+    /// `field_value` texts (absent parts are simply omitted).
+    fn add_field_region(&mut self, items: &[crate::FieldItem], parent: &str) -> String {
+        let self_ref = format!("#/field_regions/{}", self.field_regions.len());
+        self.field_regions.push(Value::Null);
+        let region_index = self.field_regions.len() - 1;
+        let mut item_refs = Vec::new();
+        for item in items {
+            item_refs.push(json!({ "$ref": self.add_field_item(item, &self_ref) }));
+        }
+        self.field_regions[region_index] = json!({
+            "self_ref": self_ref,
+            "parent": { "$ref": parent },
+            "children": item_refs,
+            "content_layer": "body",
+            "label": "field_region",
+            "prov": [],
+        });
+        self_ref
+    }
+
+    fn add_field_item(&mut self, item: &crate::FieldItem, parent: &str) -> String {
+        let self_ref = format!("#/field_items/{}", self.field_items.len());
+        self.field_items.push(Value::Null);
+        let item_index = self.field_items.len() - 1;
+        let mut child_refs = Vec::new();
+        for (label, text) in [
+            ("marker", &item.marker),
+            ("field_key", &item.key),
+            ("field_value", &item.value),
+        ] {
+            if let Some(text) = text {
+                child_refs
+                    .push(json!({ "$ref": self.add_text(label, text, &self_ref, json!({})) }));
+            }
+        }
+        self.field_items[item_index] = json!({
+            "self_ref": self_ref,
+            "parent": { "$ref": parent },
+            "children": child_refs,
+            "content_layer": "body",
+            "label": "field_item",
+            "prov": [],
+        });
+        self_ref
+    }
+
     fn add_text(&mut self, label: &str, text: &str, parent: &str, extra: Value) -> String {
         let self_ref = format!("#/texts/{}", self.texts.len());
         let raw = unescape_text(text);

crates/fleischwolf-core/src/lib.rs

@@ -16,6 +16,6 @@ mod json;
 mod labels;
 mod markdown;
 
-pub use document::{DoclingDocument, Node, PictureImage, Table};
+pub use document::{DoclingDocument, FieldItem, Node, PictureImage, Table};
 pub use labels::DocItemLabel;
 pub use markdown::{ImageMode, MarkdownStreamer};

crates/fleischwolf-core/src/markdown.rs

@@ -346,11 +346,28 @@ fn render_one(node: &Node, blocks: &mut Vec<String>, ctx: &mut Ctx) {
             blocks.push(picture_marker(image.as_ref(), ctx));
         }
         Node::Group { children, .. } => render(children, blocks, ctx),
+        Node::FieldRegion { items } => {
+            // docling renders the region container (which carries no text of its
+            // own) as a `<!-- missing-text -->` marker, then each field item the
+            // same way, followed by that item's marker/key/value as separate
+            // paragraphs.
+            blocks.push(MISSING_TEXT.to_string());
+            for item in items {
+                blocks.push(MISSING_TEXT.to_string());
+                for part in [&item.marker, &item.key, &item.value].into_iter().flatten() {
+                    blocks.push(strict_text(part, ctx.strict));
+                }
+            }
+        }
         // Handled by the run-merging branch in `render`.
         Node::ListItem { .. } => unreachable!("list items are rendered in runs"),
     }
 }
 
+/// docling's placeholder for a structural node (a field region / item) that has
+/// no text of its own.
+const MISSING_TEXT: &str = "<!-- missing-text -->";
+
 /// The Markdown for a picture under the active [`ImageMode`]; Referenced mode also
 /// records the bytes in `ctx.artifacts` for the caller to write.
 fn picture_marker(image: Option<&crate::PictureImage>, ctx: &mut Ctx) -> String {

crates/fleischwolf/Cargo.toml

@@ -15,9 +15,18 @@ rust-version.workspace = true
 # corpus by path, so they're dev-only too.
 exclude = ["tests", "benches"]
 
+[features]
+# Optional headless-browser HTML pre-render (the `--use-web-browser` flag /
+# `DocumentConverter::use_web_browser`). Off by default so the standard build
+# stays pure-Rust with no browser/runtime dependency; enable with
+# `--features web-browser`. Drives the system Chromium over the DevTools
+# protocol purely from Rust (no Node/Playwright) via `headless_chrome`.
+web-browser = ["dep:headless_chrome"]
+
 [dependencies]
 calamine = { version = "0.26", features = ["dates"] }
 csv = "1.4.0"
+headless_chrome = { version = "1.0", optional = true }
 fleischwolf-core = { path = "../fleischwolf-core", version = "0.10.0" }
 fleischwolf-pdf = { path = "../fleischwolf-pdf", version = "0.10.0" }
 image = { version = "0.25", default-features = false, features = ["png", "jpeg", "gif", "bmp", "tiff", "webp"] }