feat: dream roadmap — robustness, async, crawl intelligence, AI-ready outputs, plugins, MCP - #68

Open

obeone wants to merge 36 commits into

mainfrom

feat/dream-roadmap

Commits on May 29, 2026

chore(deps): add dev optional-dependencies (pytest, ruff, pytest-cov)
obeone
committed
b1524c9
ci: add lint and test workflow across Python 3.10-3.12
obeone
committed
20cdcde
chore: drop dead old/ and .history/ dirs, harden gitignore
obeone
committed
f9c93b2
docs(readme): align title with package name crawler-to-md
obeone
committed
4455086
refactor(db): add context manager, explicit close, WAL journal
obeone
committed
bf35030
fix(export): collapse markdown cleanup to a single pass
obeone
committed
2468e96
fix(scraper): guard crawl-loop fetch and add request timeouts
obeone
committed
670c2f1
refactor(logging): use per-module loggers, configure logging in main
obeone
committed
8c0c0b8
feat(utils): add canonicalize_url with tracking-param stripping
obeone
committed
b867968
feat(db): content-hash upsert, link depth, legacy schema migration
obeone
committed
8a5f296
feat(scraper): retry with backoff, crawl bounds, canonical link filtering
obeone
committed
bbb03df
feat(cli): wire timeout, max-retries and crawl-bound flags
obeone
committed
f49adfc
chore(deps): add httpx for async crawling
obeone
committed
782ad69
feat(scraper): add async concurrent crawl engine with parity to sync
obeone
committed
3423fd2
feat(cli): add --concurrency flag dispatching to async crawl
obeone
committed
740b017
chore(deps): add readability and render optional extras
obeone
committed
0130c49
feat(scraper): add robots, sitemap, render, extraction, auth, non-HTML ingest
obeone
committed
3ebab86
feat(cli): wire crawl-intelligence flags
obeone
committed
450aa46
chore(deps): add rag and vector optional extras
obeone
committed
840ed6a
feat(db): add get_all_pages_full with content_hash and fetched_at
obeone
committed
6e641fb
feat(export): add JSONL, llms.txt, frontmatter, chunking, vector, token totals
obeone
committed
09c56ae
feat(cli): wire AI-output flags and end-of-run summary
obeone
committed
0e5d449
feat(plugins): add Formatter/Filter/Processor/Fetcher protocols and entry-point registry
obeone
committed
a2c2d45
feat(export): route exports through the formatter registry via export_with
obeone
committed
db3ce12
chore: register first-party plugins as entry points
obeone
committed
f265d09
docs: document plugin protocols and entry-point registration
obeone
committed
4910b6f
feat(core): extract shared crawl and export orchestration
obeone
committed
10c67ae
feat(cli): add crawl/export/mcp subcommands with config file and backward-compat
obeone
committed
4e14b5f
feat(api): expose crawl() library function and package re-exports
obeone
committed
78ae369
feat(mcp): add MCP server exposing crawl and fetch_as_markdown tools
obeone
committed
e4eb71a
chore(deps): add tomli fallback and mcp optional extra
obeone
committed
7881585
docs(readme): document subcommands, new flags, extras, library API and plugins
obeone
committed
f8fd4cd

Commits on May 30, 2026