pollinate: add cactus-pollinate, a mirror-repair service

cactus-pollinate follows the Chrome MTC cosigners list, watches every
issuer's logs and every mirror's copy of them, and pushes missing
entries (via the c2sp.org/tlog-mirror write API) to any mirror that has
been lagging the log head for longer than a configured delay. CAs are
expected to push on their own; pollinate is the backstop for mirrors
they are failing to reach — including bootstrapping brand-new mirrors.

Discovery probes both layouts seen in the wild: a single log at the
bare CA base URL and mtc-tlog-profile logs at <CA prefix>/<n>. A log's
identity is the origin its checkpoint declares, including non-oid/
origins. Reads go through tlog.TileHashReader, so every hash and entry
is authenticated against the source checkpoint root without keeping a
local replica (Cloudflare's bootstrap log has 229M entries); sources
are picked freshest-first with random tie-breaks to spread read load,
and verified tiles are shared through a bounded cache.

Lag is measured against history, not the current head: the state file
(pollinate/state.json) records when the head was first seen at each
size, and a mirror is only pushed entries the head already had a full
delay window ago. Mirrors that answer "unknown origin" on the
submission API are recorded as not carrying the log and rechecked on an
interval or when the cosigners list version changes.

Issuer checkpoints are signature-verified when the key is ML-DSA-44;
mirror cosignatures on push responses are always verified, with keys
matched from the cosigners PEM bundle by key_sha256 (the SHA-256 of the
SPKI DER). Prometheus metrics (cactus_pollinate_*) and slog JSON output
follow the cactus server's conventions, including loopback-gated pprof.

Tested end-to-end against a stub mirror that verifies every pushed
subtree consistency proof, and live against a real Sunlight
witness/mirror (562-entry bootstrap push, cosignature verified, mirror
copy recomputes to the CA's signed root).

Author: mcpherrinm

CLAUDE.md

@@ -19,7 +19,7 @@ toolchain). The `Makefile` defaults `GO ?= gotip`; override with `make GO=go` on
 ## Commands
 
 ```sh
-make build              # builds bin/cactus, bin/cactus-cli, bin/cactus-keygen
+make build              # builds bin/cactus, bin/cactus-cli, bin/cactus-keygen, bin/cactus-pollinate
 make test               # gotip test ./...
 make test-race          # gotip test -race ./...
 make vet                # gotip vet ./...
@@ -86,6 +86,16 @@ abstraction. **`landmark/`** allocates §6.4 landmark sequences and serves `/lan
 processes — single-writer is enforced by documentation, not by code. See
 [docs/threat-model.md](docs/threat-model.md), [docs/disk-layout.md](docs/disk-layout.md).
 
+**`cactus-pollinate`** (`pollinate/` + `cmd/cactus-pollinate`) is a separate service, not
+part of the cactus server: it follows the Chrome MTC cosigners list
+(`specs/cosigners_schema.json`; live data at
+https://www.gstatic.com/mtcs/cosigners/v1/{cosigners.json,cosigners.pem}), polls every
+issuer's logs and every mirror's copy, and pushes missing entries to mirrors that stay
+behind the log head longer than a configured delay — reusing `mirrorpush` for the write
+side and `tlog.TileHashReader`-authenticated remote tile reads (no local log replica) for
+the read side. See the README's "Keeping mirrors in sync" section and
+`config-example-pollinate.json`.
+
 ## IDs are derived, not independent
 
 The **CA ID** (`ca_cosigner.id`, e.g. `44363.47.1.99`, the arcs *below* the `1.3.6.1.4.1` enterprise base) is load-bearing: draft

Makefile

@@ -11,6 +11,7 @@ build:
 	$(GO) build -o $(BIN_DIR)/cactus ./cmd/cactus
 	$(GO) build -o $(BIN_DIR)/cactus-cli ./cmd/cactus-cli
 	$(GO) build -o $(BIN_DIR)/cactus-keygen ./cmd/cactus-keygen
+	$(GO) build -o $(BIN_DIR)/cactus-pollinate ./cmd/cactus-pollinate
 
 test:
 	$(GO) test ./...

README.md

@@ -28,10 +28,11 @@ operate it, and where to look in the code.
 6. [Issuing your first cert](#issuing-your-first-cert)
 7. [Verifying certs with the CLI](#verifying-certs-with-the-cli)
 8. [Configuration reference](#configuration-reference)
-9. [Observability](#observability)
-10. [Layout of the codebase](#layout-of-the-codebase)
-11. [Tests](#tests)
-12. [Status](#status)
+9. [Keeping mirrors in sync: cactus-pollinate](#keeping-mirrors-in-sync-cactus-pollinate)
+10. [Observability](#observability)
+11. [Layout of the codebase](#layout-of-the-codebase)
+12. [Tests](#tests)
+13. [Status](#status)
 
 ---
 
@@ -406,6 +407,66 @@ With no targets configured the subsystem is entirely inert.
 
 ---
 
+## Keeping mirrors in sync: cactus-pollinate
+
+`cactus-pollinate` is a standalone service that repairs desynchronised
+mirrors across a whole MTC ecosystem. Point it at the Chrome MTC
+cosigners list and it follows every issuer's logs and every mirror's
+copy of them; when a mirror has been lagging a log head for longer than
+a configurable delay, it reads the missing entries from a healthy
+source (the CA's log or another mirror, load-balanced) and pushes them
+over the c2sp.org/tlog-mirror write API — the same `add-checkpoint` /
+`add-entries` flow cactus's own `mirror_push` uses.
+
+```sh
+./bin/cactus-pollinate -config config-example-pollinate.json
+```
+
+Design points:
+
+- **Inputs.** `cosigners.list`/`cosigners.keys` accept URLs or file
+  paths; the defaults are Chrome's published
+  [`cosigners.json`](https://www.gstatic.com/mtcs/cosigners/v1/cosigners.json)
+  and [`cosigners.pem`](https://www.gstatic.com/mtcs/cosigners/v1/cosigners.pem)
+  (schema in `specs/cosigners_schema.json`). Signers whose current state
+  is `REMOVED` are skipped; per-mirror overrides in `mirrors[]` can set
+  a distinct `submission_prefix` (default: the mirror's monitoring
+  `base_url`, which tlog-mirror permits) or `disable` a mirror.
+- **Log discovery.** Each issuer's `base_url` is probed at
+  `<base_url>/checkpoint` and `<base_url>/<n>/checkpoint` for
+  `n ≤ discovery.max_log_number`, since both single-log CAs and
+  mtc-tlog-profile CAs exist in the wild. A log's identity is the origin
+  its checkpoint declares — including non-`oid/` origins.
+- **The delay window (`push_delay_ms`).** CAs are expected to push on
+  their own, so pollinate records a history of log-head sizes and only
+  pushes to a mirror that is missing entries the head already had a full
+  delay window ago. After a restart it waits out one window before its
+  first push.
+- **Trust.** All reads go through `tlog.TileHashReader`, so every hash
+  and entry is authenticated against the source's checkpoint root before
+  being forwarded; checkpoints are additionally verified against the
+  issuer's key when it is ML-DSA-44. Nothing needs to *trust* pollinate
+  either — the receiving mirror re-verifies the log signature and every
+  subtree consistency proof.
+- **Coverage tracking.** Not every mirror carries every log: a mirror
+  whose submission API answers "unknown origin" is recorded as not
+  carrying that log and left alone until `not_carried_recheck_ms`
+  elapses or the cosigners list version changes.
+- **State.** Everything learned (logs, head history, per-mirror sizes
+  and verdicts) is a JSON file at `<data_dir>/pollinate/state.json`;
+  resumable upload positions live next to it under
+  `<data_dir>/mirrorpush/`.
+
+Metrics on `metrics.listen` (default `127.0.0.1:14091`), all prefixed
+`cactus_pollinate_`: log head and per-mirror sizes and lag
+(`log_head_size`, `mirror_size`, `mirror_lag_entries`,
+`mirror_carries`), push outcomes (`pushes_total{mirror,result}`,
+`pushed_entries_total`), read distribution (`source_reads_total`),
+sweep and error counters, and cosigners-list freshness. Logs are the
+same slog JSON as cactus.
+
+---
+
 ## Observability
 
 cactus emits structured JSON logs (slog) on stdout. Every line carries
@@ -437,12 +498,15 @@ cactus/
 ├── cmd/
 │   ├── cactus/         main server binary
 │   ├── cactus-cli/     debugging client (tree show, entry, cert verify, prove)
-│   └── cactus-keygen/  cosigner seed generator (-pub / -vkey / -from-vkey)
+│   ├── cactus-keygen/  cosigner seed generator (-pub / -vkey / -from-vkey)
+│   └── cactus-pollinate/ mirror-repair service (see section above)
 ├── acme/      RFC 8555 ACME server with §9 extensions
 ├── ca/        Issuer (CSR → X.509 cert via id-alg-mtcProof)
 ├── cert/      TBSCertificateLogEntry, MTCProof, CosignedMessage,
 │              CertificatePropertyList, multi-mirror sign-subtree client
 ├── mirrorpush/ c2sp.org/tlog-mirror push client (add-checkpoint, add-entries)
+├── pollinate/ mirror-repair service: follows the Chrome cosigners list,
+│              detects lagging mirrors, replays missing entries to them
 ├── landmark/  §6.4 landmark sequence allocator + /landmarks handler
 ├── log/       issuance log (single-writer, signed checkpoints + subtrees)
 ├── signer/    cosigner abstraction (ML-DSA-44/65/87, Go 1.27+)

cmd/cactus-pollinate/main.go

@@ -0,0 +1,139 @@
+// cactus-pollinate keeps MTC mirrors in sync with the issuance logs
+// they carry.
+//
+// Usage:
+//
+//	cactus-pollinate -config /path/to/pollinate.json
+//
+// It follows the Chrome MTC cosigners list, polls every issuer's logs
+// and every mirror's copy of them, and pushes missing entries (via the
+// c2sp.org/tlog-mirror write API) to any mirror that has been lagging
+// the log head for longer than the configured delay. CAs are expected
+// to push on their own; pollinate is the backstop for mirrors they are
+// failing to reach.
+package main
+
+import (
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"log/slog"
+	"net"
+	"net/http"
+	"net/http/pprof"
+	"os"
+	"os/signal"
+	"syscall"
+	"time"
+
+	"github.com/letsencrypt/cactus/logging"
+	"github.com/letsencrypt/cactus/pollinate"
+)
+
+var version = "dev"
+
+func main() {
+	configPath := flag.String("config", "pollinate.json", "path to JSON config file")
+	showVersion := flag.Bool("version", false, "print version and exit")
+	flag.Parse()
+
+	if *showVersion {
+		fmt.Println("cactus-pollinate", version)
+		return
+	}
+
+	cfg, err := pollinate.LoadConfig(*configPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "cactus-pollinate: load config: %v\n", err)
+		os.Exit(1)
+	}
+
+	logger := logging.New(os.Stdout, cfg.LogLevel)
+	slog.SetDefault(logger)
+	logger.Info("starting", "version", version, "config", *configPath,
+		"data_dir", cfg.DataDir, "cosigners", cfg.Cosigners.List,
+		"push_delay", cfg.PushDelay())
+
+	if err := run(cfg, logger); err != nil {
+		logger.Error("fatal", "err", err)
+		os.Exit(1)
+	}
+}
+
+func run(cfg pollinate.Config, logger *slog.Logger) error {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	m := pollinate.NewMetrics()
+	svc, err := pollinate.New(cfg, logger, m)
+	if err != nil {
+		return err
+	}
+
+	// Metrics + pprof, gated to loopback listeners exactly as in cactus:
+	// profiles can leak memory contents and burn CPU on demand.
+	metricsMux := http.NewServeMux()
+	metricsMux.Handle("/metrics", m.Handler())
+	if listenIsLoopback(cfg.Metrics.Listen) {
+		metricsMux.HandleFunc("/debug/pprof/", pprof.Index)
+		metricsMux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
+		metricsMux.HandleFunc("/debug/pprof/profile", pprof.Profile)
+		metricsMux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
+		metricsMux.HandleFunc("/debug/pprof/trace", pprof.Trace)
+	} else {
+		logger.Warn("metrics listener is not loopback; /debug/pprof disabled",
+			"listen", cfg.Metrics.Listen)
+	}
+	metricsHTTP := &http.Server{
+		Addr:              cfg.Metrics.Listen,
+		Handler:           metricsMux,
+		ReadHeaderTimeout: 5 * time.Second,
+		ReadTimeout:       30 * time.Second,
+		WriteTimeout:      5 * time.Minute, // pprof profiles stream for a while
+		IdleTimeout:       120 * time.Second,
+		MaxHeaderBytes:    16 * 1024,
+	}
+	srvErr := make(chan error, 1)
+	go func() {
+		logger.Info("listening", "name", "metrics", "addr", metricsHTTP.Addr)
+		if err := metricsHTTP.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
+			srvErr <- fmt.Errorf("metrics listener: %w", err)
+		}
+	}()
+
+	go svc.Run(ctx)
+
+	sigCh := make(chan os.Signal, 1)
+	signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
+	var runErr error
+	select {
+	case sig := <-sigCh:
+		logger.Info("shutting down", "signal", sig.String())
+	case err := <-srvErr:
+		logger.Error("listener failed, shutting down", "err", err)
+		runErr = err
+	}
+	cancel()
+
+	shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer shutdownCancel()
+	_ = metricsHTTP.Shutdown(shutdownCtx)
+	return runErr
+}
+
+// listenIsLoopback reports whether addr is bound to a loopback host.
+// Bare ":<port>" and "0.0.0.0:<port>" are NOT loopback.
+func listenIsLoopback(addr string) bool {
+	host, _, err := net.SplitHostPort(addr)
+	if err != nil || host == "" {
+		return false
+	}
+	if host == "localhost" {
+		return true
+	}
+	if ip := net.ParseIP(host); ip != nil {
+		return ip.IsLoopback()
+	}
+	return false
+}

config-example-pollinate.json

@@ -0,0 +1,23 @@
+{
+  "data_dir": "/tmp/cactus-pollinate-data",
+  "cosigners": {
+    "list": "https://www.gstatic.com/mtcs/cosigners/v1/cosigners.json",
+    "keys": "https://www.gstatic.com/mtcs/cosigners/v1/cosigners.pem",
+    "refresh_ms": 900000
+  },
+  "poll_interval_ms": 60000,
+  "push_delay_ms": 600000,
+  "discovery": {
+    "max_log_number": 8,
+    "interval_ms": 900000
+  },
+  "not_carried_recheck_ms": 21600000,
+  "request_timeout_ms": 30000,
+  "push_timeout_ms": 300000,
+  "max_concurrent_pushes": 4,
+  "mirrors": [],
+  "metrics": {
+    "listen": "127.0.0.1:14091"
+  },
+  "log_level": "info"
+}