feat(pipeline): unified dedup→compress→summarize pipeline (#4)

[Siddhant-K-code]

Closes #4

Summary

Adds a unified pipeline that chains dedup → compress → summarize in a single call, with per-stage stats. Exposed as both a CLI command and an HTTP endpoint.

pkg/pipeline

• Runner.Run(ctx, chunks, Options) — executes enabled stages in order
• Each stage independently enable/disable via Options
• Stats includes per-stage: input/output tokens, reduction ratio, latency
• DefaultOptions: dedup + compress enabled; summarize opt-in

distill pipeline CLI

# From stdin
echo '[{"id":"1","text":"..."}]' | distill pipeline

# From file with stats
distill pipeline --input chunks.json --output optimised.json --stats

# Tune stages
distill pipeline --dedup-threshold 0.2 --compress-ratio 0.4
distill pipeline --no-compress --summarize --summarize-max-tokens 2000

POST /v1/pipeline

{
  "chunks": [...],
  "options": {
    "dedup":     {"enabled": true, "threshold": 0.15},
    "compress":  {"enabled": true, "target_reduction": 0.5},
    "summarize": {"enabled": false}
  }
}

Response includes chunks and stats with per-stage breakdown.

Files

• pkg/pipeline/pipeline.go + pipeline_test.go (9 tests)
• cmd/pipeline.go — CLI command
• cmd/api_pipeline.go — HTTP handlers (also wires batch routes)
• cmd/api.go — registers /v1/pipeline and /v1/batch/*