A high-performance, pure-Python AsciiDoc parser built with Lark, designed for compatibility with the official AsciiDoc specification and the TCK.
-
Documentation: https://webmaven.github.io/asciidoctrine/
-
PyPI Package: https://pypi.org/project/asciidoctrine/
The Python ecosystem has long lacked a modern, maintainable, and specification-compliant AsciiDoc parser. Existing tools are often port-based or rely on regex-heavy implementations that struggle with the complex, context-sensitive nature of AsciiDoc.
AsciiDoctrine is built from the ground up to provide:
-
Spec Alignment: Strict adherence to the upcoming official AsciiDoc Language Specification.
-
First-Class AST: A structured, type-safe Abstract Syntax Tree that makes building renderers and static analysis tools a breeze.
-
Performance: Leveraging the Lark parsing engine for efficient processing of large documents.
The parser operates in a multi-pass pipeline to handle the inherent complexity of AsciiDoc:
graph LR
A[Source] --> B(Lark Parser)
B --> C[Concrete Syntax Tree]
C --> D(Transformer)
D --> E[Structured AST]
E --> F(Semantic Passes)
F --> G[Resolved ASG]
-
AST (Abstract Syntax Tree): Represented in
nodes.py, this is a structural tree of the document elements. -
ASG (Abstract Semantic Graph): The final resolved state where attributes, cross-references, and includes are fully processed.
-
Lark Parsing Engine: We use Lark because it supports multiple parsing algorithms (Earley, LALR) and has an experimental PEG mode. This allow us to handle the context-sensitive nature of AsciiDoc without the maintenance nightmare of large regex collections.
-
Two-Pass Pipeline: Handling attributes and includes requires knowing the state of the whole document. Our multi-pass approach ensures that we can resolve semantic details (like cross-references) correctly.
-
Pure Python: Zero C-extensions means easy installation on all platforms including Pyodide and WebAssembly.
End-users can install the official, stable release of AsciiDoctrine from PyPI:
pip install asciidoctrineFor developers and contributors, the recommended "happy path" is to set up a local virtual environment and install AsciiDoctrine in editable development mode:
# 1. Clone the repository
git clone https://github.com/webmaven/asciidoctrine.git
cd asciidoctrine
# 2. Create and activate a Python virtual environment
python3 -m venv venv
source venv/bin/activate
# 3. Install in development mode with test and documentation dependencies
pip install -e ".[test,docs]"[!NOTE] This package is currently under active development in tandem with the official TCK integration.
AsciiDoctrine implements a two-pass resolution pipeline. First, parse the raw source to a syntax-level AST. Then, resolve the AST to a semantic, queryable Abstract Semantic Graph (ASG):
from asciidoctrine import parse_to_ast
from asciidoctrine.resolver import ASGResolver
source = """
== Section Title
This is a *bold* word in a paragraph.
"""
# 1. Parse raw source to syntax-level AST
ast = parse_to_ast(source)
# 2. Resolve to semantic ASG (resolves attributes, includes, and filters comments)
resolver = ASGResolver(ast)
asg = resolver.resolve(ast)
# Iterate through sections in the ASG blocks list
for block in asg.get("blocks", []):
if block.get("name") == "section":
# The title is a list of inline nodes in the ASG schema
title_nodes = block.get("title", [])
title_text = "".join(node.get("value", "") for node in title_nodes if node.get("name") == "text")
print(f"Found section: {title_text}")Calling asg.to_dict() yields a structured, spec-compliant representation matching the official AsciiDoc Language ASG schema:
{
"name": "document",
"type": "block",
"blocks": [
{
"name": "section",
"type": "block",
"level": 1,
"title": {
"name": "title",
"type": "inline",
"inlines": [
{ "name": "text", "type": "string", "value": "Section Title" }
]
},
"blocks": [
{
"name": "paragraph",
"type": "block",
"inlines": [
{ "name": "text", "type": "string", "value": "This is a " },
{
"name": "span",
"type": "inline",
"variant": "strong",
"form": "constrained",
"inlines": [
{ "name": "text", "type": "string", "value": "bold" }
]
},
{ "name": "text", "type": "string", "value": " word in a paragraph." }
]
}
]
}
],
"attributes": {}
}The path to 1:1 syntactic specification compliance is tracked through the following phases:
| Phase | Focus | Status |
|---|---|---|
0 |
Foundations: PEG/Earley grammar, structured AST, and TCK test harness. |
✅ |
1 |
Advanced Blocks: Admonitions ✅, Sidebars ✅, Source blocks ✅, Example blocks ✅, and Open blocks ✅. |
✅ |
2 |
Document Infra: Headers ✅, dynamic attribute resolution ✅, and multi-file includes ✅. |
✅ |
3 |
Tables & Description Lists: Nested/mixed description lists, checklists ✅, and advanced table formatting/spans ✅. |
✅ |
4 |
Testing & Developer Experience: Callout stripping, strict AST syntax auditing ✅, and precise source location coordinate tracking ✅. |
✅ |
5 |
Structural Resolution & SSG: TOC outline extraction ✅, cross-references ( |
✅ |
6 |
Sphinx Extension Support: 100% Docutils node visitor coverage ✅, metadata alignment ✅, and permissive error recovery ✅. |
✅ |
7 |
Spec & TCK Conformance: Authoring language specification modules, ASG schema convergence, and upstream TCK contributions. |
🔄 |
asciidoctrine/
├── src/
│ └── asciidoctrine/ # Core parser logic
│ ├── grammar.lark # EBNF Grammar
│ ├── lark_parser.py # Transformer and Parser entry point
│ ├── nodes.py # AST Node definitions
│ └── __init__.py # Public API
├── examples/ # Real-world usage samples
├── tests/ # Unit and integration tests
├── pyproject.toml # Build configuration
└── README.adoc # This fileWe prioritize correctness by testing against three fronts:
1. Unit & Integration Tests: Granular tests for grammar and semantic components.
2. External Corpus (DocTest): Real-world examples from the asciidoctor-doctest corpus.
3. TCK (Technology Compatibility Kit): Direct compliance with the official AsciiDoc Language TCK suite.
Run the stable Pytest suite locally with:
pytest -k "not functional"Run the official TCK test suite with:
./run-tck.shThis project’s documentation is authored entirely in AsciiDoc, compiled using Sphinx along with the dedicated sphinx_asciidoctrine plugin, and published online at https://webmaven.github.io/asciidoctrine/!
Build and view the documentation locally:
# Compile HTML documentation
sphinx-build -b html docs/ docs/_build/html
# Open in your browser (macOS example)
open docs/_build/html/index.htmlWe welcome contributions! Please see CONTRIBUTING.adoc for detailed development workflow guidelines, TCK compliance processes, and code style.