Prometheus metrics + pprof endpoints

[KingAkeem]

Summary

Expose Prometheus metrics and pprof to operate and profile the crawler.

Motivation

• Visibility into throughput, latency, errors
• Ability to capture CPU/heap profiles

Scope

• /metrics:
  • gotor_requests_total{code,host}
  • gotor_fetch_latency_seconds histogram
  • gotor_bytes_total
  • gotor_queue_depth{host}
• /debug/pprof/* behind a flag

Acceptance Criteria

• Metrics scrapeable by Prometheus
• Short crawl shows non-zero counters and sane histograms
• pprof captures are possible during a run

Tasks

• internal/metrics package
• Register metrics in server
• Flag-gate pprof endpoints
• README: how to profile