Skip to content

Commit f8fd4cd

Browse files
obeoneobeone
committed
docs(readme): document subcommands, new flags, extras, library API and plugins
1 parent 7881585 commit f8fd4cd

1 file changed

Lines changed: 200 additions & 16 deletions

File tree

‎README.md‎

Lines changed: 200 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -34,50 +34,229 @@ crawler-to-md --url https://www.example.com
3434
- Exports data to Markdown and JSON, ready for GPT uploads. 📤
3535
- Exports each page as an individual Markdown file if `--export-individual` is used. 📝
3636
- Uses SQLite for efficient data management. 📊
37-
- Configurable via command-line arguments. ⚙️
37+
- Configurable via command-line arguments or a `crawler-to-md.toml` config file. ⚙️
3838
- Include or exclude specific HTML elements using CSS-like selectors (#id, .class, tag) during Markdown conversion. 🧩
39+
- **Robustness**: automatic retries with exponential backoff, configurable timeouts, crawl bounds. 🔁
40+
- **Concurrency**: async `httpx` engine for parallel fetching (`--concurrency N`). ⚡
41+
- **Crawl intelligence**: robots.txt compliance, sitemap seeding, boilerplate extraction, JS rendering. 🧠
42+
- **AI-ready outputs**: JSONL, llms.txt/llms-full.txt, YAML frontmatter, RAG chunks, Parquet vectors. 🤖
43+
- **MCP server**: expose crawl tools to AI agents via the Model Context Protocol. 🔌
44+
- **Library API**: use `from crawler_to_md import crawl` in your own code. 📦
45+
- **Plugin system**: extend formatters, filters, processors, and fetchers via Python entry points. 🔧
3946
- Docker support. 🐳
4047

4148
## 📋 Requirements
4249

4350
Python 3.10 or higher is required.
4451

45-
Project dependencies are managed with `pyproject.toml`. Install them with:
52+
Project dependencies are managed with `pyproject.toml`. Install the core package with:
4653

4754
```shell
48-
pip install .
55+
pip install crawler-to-md
56+
```
57+
58+
### Optional extras
59+
60+
Heavy or niche features ship as optional extras so the core stays lightweight:
61+
62+
| Extra | Install | Enables |
63+
|---|---|---|
64+
| `readability` | `pip install crawler-to-md[readability]` | Boilerplate extraction via trafilatura (`--extract readability`) |
65+
| `render` | `pip install crawler-to-md[render]` | JS rendering via Playwright (`--render`). After install run `playwright install chromium`. |
66+
| `rag` | `pip install crawler-to-md[rag]` | Token-aware RAG chunking via tiktoken (`--chunk-size`/`--chunk-overlap`); exact token counts in the run summary |
67+
| `vector` | `pip install crawler-to-md[vector]` | Parquet vector export via pyarrow (`--export-vectors`) |
68+
| `mcp` | `pip install crawler-to-md[mcp]` | MCP server (`crawler-to-md mcp`) |
69+
| `dev` | `pip install crawler-to-md[dev]` | pytest, ruff, pytest-cov |
70+
71+
Extras can be combined:
72+
73+
```shell
74+
pip install crawler-to-md[rag,vector]
75+
pip install crawler-to-md[readability,render,mcp]
4976
```
5077

5178
## 🛠 Usage
5279

53-
Start scraping with the following command:
80+
### Subcommands
81+
82+
crawler-to-md now uses subcommands. The legacy `crawler-to-md --url ...` invocation is fully preserved — when no subcommand is given the tool defaults to `crawl`.
5483

5584
```shell
56-
crawler-to-md --url <URL> [--output-folder ./output] [--cache-folder ./cache] [--overwrite-cache|-w] [--base-url <BASE_URL>] [--exclude-url <KEYWORD_IN_URL>] [--title <TITLE>] [--urls-file <URLS_FILE>] [-p <PROXY_URL>]
85+
crawler-to-md crawl --url <URL> [options] # crawl a site and export
86+
crawler-to-md export --url <URL> [options] # re-export from an existing cache (no re-crawl)
87+
crawler-to-md mcp # start the MCP server over stdio
5788
```
5889

59-
Options:
90+
### Config file
91+
92+
Place a `crawler-to-md.toml` file in your working directory and it is discovered automatically. Pass `--config path/to/file.toml` to use a different location. CLI flags always override config file values.
93+
94+
```toml
95+
# crawler-to-md.toml — example
96+
url = "https://docs.example.com"
97+
output-folder = "./docs-export"
98+
concurrency = 4
99+
max-pages = 200
100+
export-llms = true
101+
chunk-size = 512
102+
chunk-overlap = 64
103+
```
104+
105+
### `crawl` — full option reference
106+
107+
```shell
108+
crawler-to-md crawl --url <URL> [--urls-file <FILE>] [options]
109+
```
110+
111+
#### Input / output
60112

61113
- `--url`, `-u`: The starting URL. 🌍
62-
- `--urls-file`: Path to a file containing URLs to scrape, one URL per line. If '-', read from stdin. 📁
63-
- `--output-folder`, `-o`: Where to save Markdown files (default: `./output`). 📂
64-
- `--cache-folder`, `-c`: Where to store the database (default: `./cache`). 💾
65-
- `--overwrite-cache`, `-w`: Overwrite existing cache database before scraping. 🧹
114+
- `--urls-file`: Path to a file containing URLs to scrape, one URL per line. If `-`, read from stdin. 📁
115+
- `--output-folder`, `-o`: Where to save output files (default: `./output`). 📂
116+
- `--cache-folder`, `-c`: Where to store the SQLite database (default: `~/.cache/crawler-to-md`). 💾
66117
- `--base-url`, `-b`: Filter links by base URL (default: URL's base). 🔎
67-
- `--title`, `-t`: Final title of the markdown file. Defaults to the URL. 🏷️
118+
- `--title`, `-t`: Title for the output files. Defaults to the URL. 🏷️
119+
- `--config`: Path to a `crawler-to-md.toml` config file. Auto-discovered in CWD when omitted. ⚙️
120+
121+
#### Crawl control
122+
123+
- `--overwrite-cache`, `-w`: Overwrite existing cache database before scraping. 🧹
68124
- `--exclude-url`, `-e`: Exclude URLs containing this string (repeatable). ❌
69-
- `--export-individual`, `-ei`: Export each page as an individual Markdown file. 📝
70-
- `--rate-limit`, `-rl`: Maximum number of requests per minute (default: 0, no rate limit). ⏱️
71-
- `--delay`, `-d`: Delay between requests in seconds (default: 0, no delay). 🕒
125+
- `--include-url`, `-I`: Include only URLs containing this string (repeatable). ✅
126+
- `--rate-limit`, `-rl`: Maximum number of requests per minute (default: 0, no limit). ⏱️
127+
- `--delay`, `-d`: Delay between requests in seconds (default: 0). 🕒
72128
- `--proxy`, `-p`: Proxy URL for HTTP or SOCKS requests. 🌐
129+
130+
#### HTML filtering
131+
73132
- `--include`, `-i`: CSS-like selector (#id, .class, tag) to include before Markdown conversion (repeatable). ✅
74133
- `--exclude`, `-x`: CSS-like selector (#id, .class, tag) to exclude before Markdown conversion (repeatable). 🚫
75134

76-
One of the `--url` or `--urls-file` options is required.
135+
#### Robustness
136+
137+
- `--timeout`: Per-request timeout in seconds (default: `15`). ⏳
138+
- `--max-retries`: Maximum retries on transient failures — timeouts, 429, 5xx — with exponential backoff (default: `3`). 🔁
139+
- `--max-pages`: Stop after scraping this many pages (`0` = unlimited). 📏
140+
- `--max-depth`: Maximum link-discovery depth (`-1` = unlimited). 🌊
141+
- `--max-time`: Maximum wall-clock crawl time in seconds (`0` = unlimited). ⏰
142+
143+
#### Concurrency
144+
145+
- `--concurrency N`: Number of parallel fetches. `1` (default) uses the synchronous engine; `N > 1` enables the async `httpx` engine with a bounded semaphore. Per-host politeness (rate limit, delay) is preserved in async mode. ⚡
146+
147+
#### Intelligence
148+
149+
- `--ignore-robots`: Disable robots.txt compliance (robots.txt is **honored by default**). 🤖
150+
- `--user-agent`: Custom User-Agent string sent on every request. 🪪
151+
- `--sitemap`: Seed the crawl frontier from the host's `/sitemap.xml` before crawling. 🗺️
152+
- `--extract {none,readability}`: Content-extraction strategy. `readability` uses trafilatura to strip boilerplate (requires `pip install crawler-to-md[readability]`). Default: `none`. 🧹
153+
- `--render`: Fetch JS-rendered HTML via Playwright (requires `pip install crawler-to-md[render]` and `playwright install chromium`). Off by default. 🎭
154+
- `--header "Key: Value"`: Extra request header (repeatable). 📬
155+
- `--cookie "key=value"`: Request cookie (repeatable). 🍪
156+
- `--auth user:pass`: HTTP Basic authentication credentials. 🔑
157+
- `--allow-types application/pdf`: Additional MIME types to ingest via MarkItDown — e.g. `application/pdf`, `application/vnd.openxmlformats-officedocument.wordprocessingml.document` (repeatable). 📎
158+
159+
#### Output formats
160+
161+
- `--export-individual`, `-ei`: Export each page as an individual Markdown file. 📝
162+
- `--frontmatter` / `--no-frontmatter`: Prepend YAML frontmatter (url, title, fetched_at, word_count, token_count) to individual Markdown files. On by default. 🗂️
163+
- `--no-markdown`: Skip generation of the compiled Markdown file. 🚫
164+
- `--no-json`: Skip generation of the compiled JSON file. 🚫
165+
- `--export-jsonl`: Export pages as JSON Lines — one `{url, content, metadata}` record per line. 🗃️
166+
- `--export-llms`: Export `llms.txt` (page index) and `llms-full.txt` (full content) in the emerging LLM-friendly format. 🤖
167+
- `--chunk-size N`: Split pages into RAG chunks of `N` tokens (0 = disabled). Requires `pip install crawler-to-md[rag]`. 🧩
168+
- `--chunk-overlap N`: Token overlap between consecutive chunks (used when `--chunk-size > 0`). 🔗
169+
- `--export-vectors`: Export pages to a Parquet file for downstream vector indexing. Requires `pip install crawler-to-md[vector]`. 📊
170+
171+
#### Run summary
172+
173+
Every run prints an end-of-run summary to stdout:
174+
175+
```
176+
Run summary
177+
Links discovered : 142
178+
Pages scraped : 98
179+
Pages stored : 98
180+
Content bytes : 1048576
181+
Total tokens : 210340 (estimated)
182+
Duration : 34.21s
183+
```
184+
185+
Token counts are exact when the `rag` extra is installed (tiktoken), or estimated otherwise.
186+
187+
### `export` — re-export without re-crawling
188+
189+
Re-run any combination of export formats from an existing cache database without hitting the network again. Accepts all the same output flags as `crawl`.
190+
191+
```shell
192+
crawler-to-md export --url https://docs.example.com \
193+
--export-llms --export-jsonl --chunk-size 512 --chunk-overlap 64
194+
```
195+
196+
### `mcp` — MCP server
197+
198+
Expose crawl tools to AI agents and orchestrators that speak the [Model Context Protocol](https://modelcontextprotocol.io/) over stdio.
199+
200+
```shell
201+
# requires the mcp extra
202+
pip install crawler-to-md[mcp]
203+
crawler-to-md mcp
204+
```
205+
206+
Tools exposed: `crawl` and `fetch_as_markdown`.
77207

78208
### 📚 Log level
79209

80-
By default, the `WARN` level is used. You can change it with the `LOG_LEVEL` environment variable.
210+
By default, the `WARN` level is used. Change it with the `LOG_LEVEL` environment variable:
211+
212+
```shell
213+
LOG_LEVEL=INFO crawler-to-md --url https://example.com
214+
```
215+
216+
## 📦 Library API
217+
218+
Use `crawler_to_md` as a library — no CLI side effects, no `sys.argv` parsing.
219+
220+
```python
221+
from crawler_to_md import crawl
222+
223+
result = crawl("https://docs.example.com", max_pages=50, concurrency=4)
224+
225+
for page in result.pages:
226+
print(page.url, len(page.content))
227+
228+
print(f"Scraped {result.stats.pages_scraped} pages in {result.stats.duration:.1f}s")
229+
```
230+
231+
`crawl()` returns a `CrawlResult` object with:
232+
233+
- `result.pages` — list of scraped pages (url, content, metadata)
234+
- `result.stats` — aggregate run statistics (pages scraped, bytes, tokens, duration)
235+
- `result.exports` — paths of any files written (if export options are passed)
236+
237+
## 🔧 Plugin System
238+
239+
crawler-to-md exposes an entry-point-based plugin architecture for all four pipeline stages. See **[docs/plugins.md](docs/plugins.md)** for the full reference.
240+
241+
### Entry-point groups
242+
243+
| Stage | Group | Protocol method |
244+
|---|---|---|
245+
| **Formatter** | `crawler_to_md.formatters` | `export(manager, output_path, **options)` |
246+
| **Filter** | `crawler_to_md.filters` | `is_allowed(url) -> bool` |
247+
| **Processor** | `crawler_to_md.processors` | `process(html, url) -> str` |
248+
| **Fetcher** | `crawler_to_md.fetchers` | `fetch(url)` |
249+
250+
All built-in output formats ship as first-party formatters (`markdown`, `json`, `jsonl`, `llms`, `individual`, `chunks`, `vectors`) and are wired live through the registry. Register your own by declaring an entry point in your package's `pyproject.toml`:
251+
252+
```toml
253+
[project.entry-points."crawler_to_md.formatters"]
254+
my-format = "my_package.exporters:MyFormatter"
255+
```
256+
257+
Once your package is installed, the plugin is discovered automatically. See `tests/sample_plugin.py` for a worked end-to-end example.
258+
259+
> **Note**: The filter, processor, and fetcher plugin protocols and registries are defined and tested, but are not yet consumed by the crawl loop — see [Known Limitations](#-known-limitations--follow-ups).
81260
82261
## 🐳 Docker Support
83262

@@ -100,6 +279,11 @@ docker run --rm \
100279
crawler-to-md --url <URL>
101280
```
102281

282+
## ⚠️ Known Limitations / Follow-ups
283+
284+
- **Filter/processor/fetcher plugin protocols** are fully defined and tested but are not yet consumed by the crawl loop. Formatters are wired live; the other three stages are available for extension and will be integrated in a future release.
285+
- **Sitemap parsing** uses the stdlib `xml.etree.ElementTree`. For untrusted or adversarially crafted sitemaps, consider using `defusedxml` as a drop-in replacement to guard against XML DoS attacks (billion-laughs).
286+
103287
## 🤝 Contributing
104288

105289
Contributions are welcome! Feel free to submit pull requests or open issues. 🌟

0 commit comments

Comments
 (0)