Skip to content

Commit b227474

Browse files
committed
docs: update distill-page readme to describe ast compiler and blockquote limitation
1 parent 6063aab commit b227474

1 file changed

Lines changed: 8 additions & 2 deletions

File tree

agents/skills/distill-page/README.md

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,20 @@ Unlike tools like convert page DOM/HTML to markdown, `distill-page` uses Chromiu
2929
* **paulirish.com homepage**: 7.4KB (`turndown`) → **2.4KB** (`distill-page`) — **67% savings**
3030
* This saves valuable context window tokens and reduces processing costs when feeding pages to downstream LLMs.
3131
3. **Semantic Fidelity Features**:
32-
* **Page Titles**: Extracts and prepends the page title as a `# Title` header.
32+
* **Page Titles**: Extracts and prepends the page title as a standalone plain text line at the very top.
3333
* **Table Captions**: Identifies table captions (`TableData.table_name`) and renders them cleanly as bold headers.
3434
* **Ad Stripping**: Instantly ignores nodes flagged as ads by the browser model (`is_ad_related`).
3535
* **Accordions**: Detects collapsed elements and wraps them in HTML `<details>` disclosure blocks.
3636
* **Asides**: Detects sidebars and callouts and wraps them in HTML `<aside>` tags (preventing heading hierarchy breakage).
37-
* **Heading Levels**: Dynamic mapping of text size hierarchies to appropriate Markdown headers (H1-H5).
37+
* **Heading Levels**: Dynamic mapping of text size hierarchies to appropriate Markdown headers (`XL` &rarr; `# H1`, `L` &rarr; `## H2`, `M` &rarr; `### H3`, etc.).
3838

3939
## 🛠️ Implementation Details
4040

41+
### Two-Pass Compiler Architecture
42+
The parser uses a clean two-pass compiler design to decouple tree parsing from markdown serialization formatting:
43+
1. **Phase 1 (Parse to AST)**: Recursively traverses the Chromium layout annotation protobufs (`AnnotatedPageContent`) and builds a strongly-typed intermediate document AST (`ASTDocument` containing `ASTBlockNode` and `ASTInlineNode` structures).
44+
2. **Phase 2 (Serialize to Markdown)**: Traverses the AST and formats it into Markdown. This keeps formatting logic (spacing normalization, tag generation, emphasis grouping) separate from structural parsing.
45+
4146
### Core Mechanism
4247
The script uses **Playwright** to launch Chromium and establish a direct **CDP Session** to a page. It then calls the experimental `Page.getAnnotatedPageContent` method to extract the ML-annotated layout tree.
4348

@@ -54,6 +59,7 @@ The CDP method returns a base64 string containing a serialized `AnnotatedPageCon
5459

5560
* **Empty Pages**: The CDP method will fail with a protocol error on `about:blank` or pages with negligible content.
5661
* **Experimental Status**: This is an experimental CDP feature; its behavior may change or be removed in future Chromium versions.
62+
* **Blockquotes**: The Chromium layout annotator model does not recognize blockquotes and maps `<blockquote>` elements to `AX_ROLE_UNKNOWN` (181). Because of this, Page Distiller does not support blockquote structural nesting or styling (we choose not to handle it because raw DOM injection fixes are fragile and break on nested line boundaries).
5763

5864
## 🆚 Turndown vs CDP ML Extraction (Tradeoffs)
5965

0 commit comments

Comments
 (0)