You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* This saves valuable context window tokens and reduces processing costs when feeding pages to downstream LLMs.
31
31
3.**Semantic Fidelity Features**:
32
-
***Page Titles**: Extracts and prepends the page title as a `# Title` header.
32
+
***Page Titles**: Extracts and prepends the page title as a standalone plain text line at the very top.
33
33
***Table Captions**: Identifies table captions (`TableData.table_name`) and renders them cleanly as bold headers.
34
34
***Ad Stripping**: Instantly ignores nodes flagged as ads by the browser model (`is_ad_related`).
35
35
***Accordions**: Detects collapsed elements and wraps them in HTML `<details>` disclosure blocks.
36
36
***Asides**: Detects sidebars and callouts and wraps them in HTML `<aside>` tags (preventing heading hierarchy breakage).
37
-
***Heading Levels**: Dynamic mapping of text size hierarchies to appropriate Markdown headers (H1-H5).
37
+
***Heading Levels**: Dynamic mapping of text size hierarchies to appropriate Markdown headers (`XL`→`# H1`, `L`→`## H2`, `M`→`### H3`, etc.).
38
38
39
39
## 🛠️ Implementation Details
40
40
41
+
### Two-Pass Compiler Architecture
42
+
The parser uses a clean two-pass compiler design to decouple tree parsing from markdown serialization formatting:
43
+
1.**Phase 1 (Parse to AST)**: Recursively traverses the Chromium layout annotation protobufs (`AnnotatedPageContent`) and builds a strongly-typed intermediate document AST (`ASTDocument` containing `ASTBlockNode` and `ASTInlineNode` structures).
44
+
2.**Phase 2 (Serialize to Markdown)**: Traverses the AST and formats it into Markdown. This keeps formatting logic (spacing normalization, tag generation, emphasis grouping) separate from structural parsing.
45
+
41
46
### Core Mechanism
42
47
The script uses **Playwright** to launch Chromium and establish a direct **CDP Session** to a page. It then calls the experimental `Page.getAnnotatedPageContent` method to extract the ML-annotated layout tree.
43
48
@@ -54,6 +59,7 @@ The CDP method returns a base64 string containing a serialized `AnnotatedPageCon
54
59
55
60
***Empty Pages**: The CDP method will fail with a protocol error on `about:blank` or pages with negligible content.
56
61
***Experimental Status**: This is an experimental CDP feature; its behavior may change or be removed in future Chromium versions.
62
+
***Blockquotes**: The Chromium layout annotator model does not recognize blockquotes and maps `<blockquote>` elements to `AX_ROLE_UNKNOWN` (181). Because of this, Page Distiller does not support blockquote structural nesting or styling (we choose not to handle it because raw DOM injection fixes are fragile and break on nested line boundaries).
0 commit comments