Skip to content

Adopt Tower & add function-style crawler API (extract / execute / store) #81

@oiwn

Description

@oiwn

Introduce a small Tower-based core so a crawler can run as a simple function while remaining fully composable (timeouts, retries, rate-limits, tracing). Provide crawl_once(extract, exec, store) that wires three Services together; supply adapters so users can pass plain async functions/closures.

Motivation:

  • Keep ergonomics: “computation” as a function.
  • Gain reliability + reuse: Tower layers for backoff, concurrency, timeouts, tracing.
  • Easier testing: mock any stage with service_fn.
  • Backend neutrality: queue/http/store are swappable.

Tasks

  • Create capp-tower feature / crate.
  • Implement crawl_once and core error CErr.
  • Add adapters: closures → Service (service_fn).
  • Provide example: in-memory extract, reqwest exec stack, Mongo store.
  • Add tower-http tracing layer integration with tracing.
  • Integration tests: happy path, timeout, retry, DLQ handoff.
  • Bench: baseline vs layered exec (show overhead).
  • Docs: quickstart + migration notes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions