feat: dream roadmap — robustness, async, crawl intelligence, AI-ready outputs, plugins, MCP

[obeone]

Summary

End-to-end implementation of the "dream roadmap" (.omc/plans/dream-roadmap.md), executed in quality-gated phases (P0 → P3 subset). Every phase landed with tests green and ruff clean before the next began. 31 commits, atomic and conventional.

Test suite grew 40 → 115 (all green; bare CI env stays green via importorskip/monkeypatch on optional extras).

What changed, by phase

P0 — Foundation (hygiene + real defect fixes + tooling)

• Bug: crawl-loop session.get was unguarded → one network error crashed the whole crawl. Now wrapped + every request has a timeout.
• Bug: O(n²) markdown assembly (_cleanup_markdown ran per page on the whole growing doc) → now a single pass.
• Bug: DB closed via __del__ → now context manager + explicit close() + WAL.
• Bug: shared root-logger renamed by every module → per-module logging.getLogger(__name__); logging configured in main(), not at import.
• Dead old/ + .history/ removed; .gitignore hardened; CI workflow added (ruff + pytest on 3.10–3.12); dev extras declared; README title fixed.

P1 — Robustness

Retry with exponential backoff + jitter (429/5xx/Retry-After), --timeout, content-hash upsert (fetched_at, only rewrites on change), crawl bounds (--max-pages/--max-depth/--max-time), canonical URL dedup (tracking-param stripping, port/trailing-slash normalization). Legacy SQLite caches migrated additively.

P1.5 — Async concurrency

httpx.AsyncClient engine behind --concurrency N (N=1 = byte-for-byte sync path, retained). SQLite touched only from the main coroutine; network runs concurrently under a semaphore. Parity test asserts identical corpus sync vs async.

P2 — Crawl intelligence + AI-ready outputs

• Intelligence: robots.txt (default on, --ignore-robots), --user-agent, --sitemap, --extract readability (extra), --render (extra, Playwright), --header/--cookie/--auth, --allow-types (PDF/docx via MarkItDown). Applied to both sync + async paths.
• Outputs (§5): --export-jsonl, --export-llms (llms.txt + llms-full.txt), YAML frontmatter, token-aware RAG chunking (extra), token accounting, Parquet vector export (extra), end-of-run summary.

P2.5 — Plugin system

Entry-point registry + Formatter/Filter/Processor/Fetcher protocols. Formatters wired live (output byte-identical); other groups have protocols + first-party impls + an example third-party plugin proven via a test. See docs/plugins.md.

P3 (subset) — Interfaces

• Subcommands crawl|export|mcp with backward-compat (crawler-to-md --url … still works); crawler-to-md.toml config (CLI overrides file).
• Library API: from crawler_to_md import crawl(...) → CrawlResult, no CLI side effects; shared orchestration in core.py.
• MCP server (crawler-to-md mcp, extra mcp) exposing crawl and fetch_as_markdown tools.

Optional extras

readability (trafilatura), render (playwright), rag (tiktoken), vector (pyarrow), mcp, dev. Core install stays light; all gated features lazy-import with a clear error if the extra is missing.

Deferred / follow-ups (intentional, for honesty)

• filter/processor/fetcher plugin protocols exist + are tested but are not yet consumed by the live crawl loop (deferred to avoid regression risk).
• Sitemap XML parsed with stdlib ElementTree; consider defusedxml (or a size/depth guard) for untrusted input.
• HTTP service + web UI and the incremental scheduler were explicitly out of scope for this batch.

Verification

• ruff check . clean.
• pytest: 115 passed with all extras installed; bare dev env green (optional-extra tests skip/guard).
• Backward compatibility preserved: all pre-existing CLI flags and the bare --url invocation still work.

[obeone]

Please massively improve documentation!
And adapt README.md

[obeone]

Addressed both review comments on this branch:

Documentation (massive improvement):

• Rewrote README.md to the standard format: shields.io badges, a Mermaid architecture diagram, feature/flag tables, and full crawl/export/mcp references.
• Added a real docs set: docs/README.md (index), docs/configuration.md (full crawler-to-md.toml reference), docs/library.md (library API), docs/mcp.md (MCP server) — plus the existing docs/plugins.md.
• Added the missing LICENSE file (MIT, matching pyproject).
• Fixed a doc/code mismatch: CrawlResult.pages items are dicts, so the README example now uses page["url"].

CodeQL — workflow token permissions:

• Added an explicit permissions: contents: read block to .github/workflows/ci.yaml.

All pushed to feat/dream-roadmap.