Introduce a small Tower-based core so a crawler can run as a simple function while remaining fully composable (timeouts, retries, rate-limits, tracing). Provide crawl_once(extract, exec, store) that wires three Services together; supply adapters so users can pass plain async functions/closures.
Motivation:
- Keep ergonomics: “computation” as a function.
- Gain reliability + reuse: Tower layers for backoff, concurrency, timeouts, tracing.
- Easier testing: mock any stage with service_fn.
- Backend neutrality: queue/http/store are swappable.
Tasks
Introduce a small Tower-based core so a crawler can run as a simple function while remaining fully composable (timeouts, retries, rate-limits, tracing). Provide crawl_once(extract, exec, store) that wires three Services together; supply adapters so users can pass plain async functions/closures.
Motivation:
Tasks