refactor: add support for postgres store(s) (#28)

* refactor: add support for postgres store(s)

closes #27

* fix: introduce MigratedPool type for proper startup

* refactor: remove job queue from postgres agent store

Author: frrist

CLAUDE.md

@@ -42,9 +42,11 @@ Sprue is the upload coordination service for Storacha local development. It rout
 
 **Stores (pkg/store/)**
 - Each domain has its own store interface in `pkg/store/<domain>/`
-- Each store has two implementations: AWS (DynamoDB/S3) in `<domain>/aws/` and in-memory in `<domain>/memory/`
+- Each store has three implementations: AWS (DynamoDB/S3) in `<domain>/aws/`, PostgreSQL (+ S3 for blob payloads) in `<domain>/postgres/`, and in-memory in `<domain>/memory/`
 - Store interfaces: `agent.Store`, `blob_registry.Store`, `consumer.Store`, `customer.Store`, `delegation.Store`, `metrics.Store`, `replica.Store`, `revocation.Store`, `space_diff.Store`, `storage_provider.Store`, `subscription.Store`, `upload.Store`
-- AWS stores are wired in `internal/fx/store/aws/provider.go`, memory stores in `internal/fx/store/memory/provider.go`
+- Backends are wired in `internal/fx/store/<backend>/provider.go` (aws, postgres, memory)
+- Backend selection is driven by `storage.type` in config (`memory` | `postgres` | `aws`; default `postgres`). Per-backend settings live under `storage.postgres`, `storage.dynamodb`, and `storage.s3`.
+- Postgres schema is managed by goose migrations in `internal/migrations/sql/`, embedded and applied on startup. Set `storage.postgres.skip_migrations: true` to disable.
 
 **Services (pkg/)**
 - `provisioning`: Manages space provisioning (consumers + subscriptions)
@@ -66,18 +68,21 @@ Sprue is the upload coordination service for Storacha local development. It rout
 
 ### Configuration
 
-Configuration via YAML file or environment variables with `UPLOAD_` prefix:
-- `UPLOAD_SERVER_HOST`, `UPLOAD_SERVER_PORT`
-- `UPLOAD_IDENTITY_KEY_FILE`, `UPLOAD_IDENTITY_PRIVATE_KEY`, `UPLOAD_IDENTITY_SERVICE_DID`
-- `UPLOAD_PIRI_ENDPOINT`, `UPLOAD_INDEXER_ENDPOINT`
-- `UPLOAD_DYNAMODB_*` for DynamoDB settings
-
-Legacy env vars without prefix (e.g., `HOST`, `PORT`, `KEY_FILE`) also supported.
+Configuration via YAML file or environment variables with `SPRUE_` prefix:
+- `SPRUE_STORAGE_TYPE` — selects the store backend (`memory`, `postgres`, `aws`; default `postgres`)
+- `SPRUE_SERVER_HOST`, `SPRUE_SERVER_PORT`
+- `SPRUE_IDENTITY_KEY_FILE`, `SPRUE_IDENTITY_PRIVATE_KEY`, `SPRUE_IDENTITY_SERVICE_DID`
+- `SPRUE_INDEXER_ENDPOINT`
+- `SPRUE_STORAGE_POSTGRES_DSN`, `SPRUE_STORAGE_POSTGRES_MAX_CONNS`, `SPRUE_STORAGE_POSTGRES_SKIP_MIGRATIONS`
+- `SPRUE_STORAGE_DYNAMODB_*` for DynamoDB settings (AWS backend)
+- `SPRUE_STORAGE_S3_*` for S3/MinIO settings
 
 ### Key Dependencies
 
 - **go-ucanto**: UCAN RPC framework for capability-based authorization
 - **go-libstoracha**: Storacha capability definitions (blob, space, upload, etc.)
 - **echo/v4**: HTTP server framework
-- **aws-sdk-go-v2**: DynamoDB client
+- **aws-sdk-go-v2**: DynamoDB + S3 client
+- **jackc/pgx/v5**: PostgreSQL driver
+- **pressly/goose/v3**: SQL schema migrations
 - **viper/cobra**: Configuration and CLI

Makefile

@@ -2,6 +2,7 @@ VERSION=$(shell awk -F'"' '/"version":/ {print $$4}' version.json)
 COMMIT=$(shell git rev-parse --short HEAD)
 DATE=$(shell date -u -Iseconds)
 GOFLAGS=-ldflags="-X github.com/storacha/sprue/pkg/build.version=$(VERSION) -X github.com/storacha/sprue/pkg/build.Commit=$(COMMIT) -X github.com/storacha/sprue/pkg/build.Date=$(DATE) -X github.com/storacha/sprue/pkg/build.BuiltBy=make"
+DOCKER := $(shell which docker)
 
 .PHONY: all build test lint clean docker-build
 
@@ -35,4 +36,4 @@ clean:
 	rm -f ./sprue
 
 docker-build:
-	docker build -t sprue:latest .
+	$(DOCKER) build -t sprue:latest .

README.md

@@ -1,6 +1,30 @@
 # Sprue
 
-The Storacha upload service in Go.
+The Forge upload service in Go (formerly the Storacha upload service).
+
+## Running locally
+
+The repo ships a `docker-compose.yaml` that brings up sprue alongside
+PostgreSQL and MinIO for self-hosted development:
+
+```bash
+docker compose up -d postgres minio
+SPRUE_STORAGE_POSTGRES_DSN="postgres://sprue:sprue@localhost:5432/sprue?sslmode=disable" \
+  ./sprue serve
+```
+
+Postgres is the default store backend, so no extra flag is required.
+
+## Store backends
+
+Sprue supports three store backends, selected by
+`storage.type` (or `SPRUE_STORAGE_TYPE`; defaults to `postgres`):
+
+- `memory` — in-process only; all data is lost on restart. Dev/test only.
+- `postgres` — PostgreSQL for metadata + S3-compatible storage (MinIO, Ceph, AWS S3)
+  for blob payloads. Schema is managed by goose migrations embedded in
+  `internal/migrations/sql/` and applied on startup.
+- `aws` — DynamoDB for metadata + S3 for blob payloads.
 
 ## Notes
 

config.example.yaml

@@ -1,12 +1,13 @@
 # Sprue Configuration
 #
 # This file documents all configuration options. Copy to config.yaml and customize.
-# All values can also be set via environment variables with UPLOAD_ prefix.
+# All values can also be set via environment variables with SPRUE_ prefix.
 # Environment variables take precedence over config file values.
 #
 # Examples:
-#   server.host -> UPLOAD_SERVER_HOST
-#   dynamodb.endpoint -> UPLOAD_DYNAMODB_ENDPOINT
+#   server.host          -> SPRUE_SERVER_HOST
+#   storage.type         -> SPRUE_STORAGE_TYPE
+#   storage.postgres.dsn -> SPRUE_STORAGE_POSTGRES_DSN
 #
 
 # Deployment configuration
@@ -20,11 +21,6 @@ deployment:
   # includes the original blob that was uploaded, so only values above 1 will
   # allow users to have multiple copies of their data.
   max_replicas: 3
-  # Indicates whether to use in-memory stores instead of DynamoDB/S3. All data 
-  # will be lost on service restart when this is true, so it should only be used
-  # for development or testing. It overrides all other store-related config when
-  # true.
-  in_memory_stores: false
 
 server:
   # Host address to bind the HTTP server to
@@ -63,33 +59,51 @@ mailer:
   # Secret for CRAMMD5 SMTP authentication
   smtp_auth_secret: ""
 
-dynamodb:
-  # DynamoDB endpoint (for local development with DynamoDB Local)
-  endpoint: "http://dynamodb-local:8000"
-  # AWS region for DynamoDB
-  region: "us-west-1"
-  agent_index_table: "agent-index"
-  blob_registry_table: "blob-registry"
-  consumer_table: "consumer"
-  customer_table: "customer"
-  delegation_table: "delegation"
-  space_metrics_table: "space-metrics"
-  admin_metrics_table: "admin-metrics"
-  replica_table: "replica"
-  revocation_table: "revocation"
-  storage_provider_table: "storage-provider"
-  subscription_table: "subscription"
-  space_diff_table: "space-diff"
-  upload_table: "upload"
+# Storage backend selection and per-backend configuration.
+storage:
+  # Selects the backend. Valid values: "memory", "postgres", "aws". Default "postgres".
+  #  - memory:   in-process stores — all data lost on restart; dev/test only.
+  #  - postgres: PostgreSQL for metadata + S3-compatible storage for blob payloads.
+  #  - aws:      DynamoDB for metadata + S3 for blob payloads.
+  type: "postgres"
 
-s3:
-  # S3 endpoint (for local development with MinIO)
-  endpoint: "http://minio:9000"
-  # S3 region
-  region: "us-west-1"
-  agent_message_bucket: "agent-message"
-  delegation_bucket: "delegation"
-  upload_shards_bucket: "upload-shards"
+  postgres:
+    # libpq-style connection string (used when storage.type is "postgres").
+    dsn: "postgres://sprue:sprue@postgres:5432/sprue?sslmode=disable"
+    # Maximum number of pool connections. 0 uses the pgx default.
+    max_conns: 10
+    # Minimum idle connections to keep warm.
+    min_conns: 0
+    # Set to true to skip goose migrations at startup (they run by default).
+    skip_migrations: false
+
+  dynamodb:
+    # DynamoDB endpoint (for local development with DynamoDB Local).
+    endpoint: "http://dynamodb-local:8000"
+    # AWS region for DynamoDB.
+    region: "us-west-1"
+    agent_index_table: "agent-index"
+    blob_registry_table: "blob-registry"
+    consumer_table: "consumer"
+    customer_table: "customer"
+    delegation_table: "delegation"
+    space_metrics_table: "space-metrics"
+    admin_metrics_table: "admin-metrics"
+    replica_table: "replica"
+    revocation_table: "revocation"
+    storage_provider_table: "storage-provider"
+    subscription_table: "subscription"
+    space_diff_table: "space-diff"
+    upload_table: "upload"
+
+  s3:
+    # S3 endpoint (for local development with MinIO).
+    endpoint: "http://minio:9000"
+    # S3 region.
+    region: "us-west-1"
+    agent_message_bucket: "agent-message"
+    delegation_bucket: "delegation"
+    upload_shards_bucket: "upload-shards"
 
 log:
   # Log level: debug, info, warn, error

go.mod

@@ -13,23 +13,41 @@ require (
 	github.com/ipfs/go-cid v0.6.0
 	github.com/ipfs/go-log/v2 v2.9.0
 	github.com/ipld/go-ipld-prime v0.21.1-0.20240917223228-6148356a4c2e
+	github.com/jackc/pgx/v5 v5.9.1
 	github.com/labstack/echo/v4 v4.14.0
 	github.com/multiformats/go-multiaddr v0.16.0
 	github.com/multiformats/go-multibase v0.2.0
 	github.com/multiformats/go-multihash v0.2.3
 	github.com/olekukonko/tablewriter v0.0.5
+	github.com/pressly/goose/v3 v3.27.0
 	github.com/spf13/cobra v1.10.2
 	github.com/spf13/viper v1.21.0
 	github.com/storacha/go-libstoracha v0.7.5
 	github.com/storacha/go-ucanto v0.7.2
 	github.com/stretchr/testify v1.11.1
-	github.com/testcontainers/testcontainers-go v0.41.0
+	github.com/testcontainers/testcontainers-go v0.42.0
 	github.com/testcontainers/testcontainers-go/modules/dynamodb v0.41.0
 	github.com/testcontainers/testcontainers-go/modules/minio v0.40.0
+	github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
 	go.uber.org/fx v1.24.0
 	go.uber.org/zap v1.27.0
 )
 
+require (
+	github.com/jackc/pgpassfile v1.0.0 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
+	github.com/jackc/puddle/v2 v2.2.2 // indirect
+	github.com/mfridman/interpolate v0.0.2 // indirect
+	github.com/moby/moby/api v1.54.1 // indirect
+	github.com/moby/moby/client v0.4.0 // indirect
+	github.com/moby/sys/atomicwriter v0.1.0 // indirect
+	github.com/sethvargo/go-retry v0.3.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.43.0 // indirect
+	go.opentelemetry.io/otel/sdk/metric v1.43.0 // indirect
+	golang.org/x/sync v0.20.0 // indirect
+)
+
 require (
 	dario.cat/mergo v1.0.2 // indirect
 	github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
@@ -59,7 +77,6 @@ require (
 	github.com/containerd/log v0.1.0 // indirect
 	github.com/containerd/platforms v0.2.1 // indirect
 	github.com/cpuguy83/dockercfg v0.3.2 // indirect
-	github.com/creack/pty v1.1.24 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
 	github.com/distribution/reference v0.6.0 // indirect
@@ -78,7 +95,6 @@ require (
 	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/gobwas/glob v0.2.3 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
-	github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
 	github.com/hashicorp/golang-lru v1.0.2 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/ipfs/bbloom v0.0.4 // indirect
@@ -99,7 +115,7 @@ require (
 	github.com/ipld/go-car v0.6.2 // indirect
 	github.com/ipld/go-codec-dagpb v1.6.0 // indirect
 	github.com/ipni/go-libipni v0.6.18 // indirect
-	github.com/klauspost/compress v1.18.2 // indirect
+	github.com/klauspost/compress v1.18.5 // indirect
 	github.com/klauspost/cpuid/v2 v2.2.10 // indirect
 	github.com/labstack/gommon v0.4.2 // indirect
 	github.com/libp2p/go-buffer-pool v0.1.0 // indirect
@@ -112,12 +128,11 @@ require (
 	github.com/minio/sha256-simd v1.0.1 // indirect
 	github.com/moby/docker-image-spec v1.3.1 // indirect
 	github.com/moby/go-archive v0.2.0 // indirect
-	github.com/moby/patternmatcher v0.6.0 // indirect
+	github.com/moby/patternmatcher v0.6.1 // indirect
 	github.com/moby/sys/sequential v0.6.0 // indirect
 	github.com/moby/sys/user v0.4.0 // indirect
 	github.com/moby/sys/userns v0.1.0 // indirect
 	github.com/moby/term v0.5.2 // indirect
-	github.com/morikuni/aec v1.0.0 // indirect
 	github.com/mr-tron/base58 v1.2.0 // indirect
 	github.com/multiformats/go-base32 v0.1.0 // indirect
 	github.com/multiformats/go-base36 v0.2.0 // indirect
@@ -136,8 +151,8 @@ require (
 	github.com/polydawn/refmt v0.89.1-0.20231129105047-37766d95467a // indirect
 	github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
 	github.com/sagikazarmark/locafero v0.11.0 // indirect
-	github.com/shirou/gopsutil/v4 v4.26.2 // indirect
-	github.com/sirupsen/logrus v1.9.3 // indirect
+	github.com/shirou/gopsutil/v4 v4.26.3 // indirect
+	github.com/sirupsen/logrus v1.9.4 // indirect
 	github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
 	github.com/spaolacci/murmur3 v1.1.0 // indirect
 	github.com/spf13/afero v1.15.0 // indirect
@@ -153,24 +168,22 @@ require (
 	github.com/yusufpapurcu/wmi v1.2.4 // indirect
 	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
 	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
-	go.opentelemetry.io/otel v1.42.0 // indirect
-	go.opentelemetry.io/otel/metric v1.42.0 // indirect
-	go.opentelemetry.io/otel/trace v1.42.0 // indirect
+	go.opentelemetry.io/otel v1.43.0 // indirect
+	go.opentelemetry.io/otel/metric v1.43.0 // indirect
+	go.opentelemetry.io/otel/trace v1.43.0 // indirect
 	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/dig v1.19.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/crypto v0.48.0 // indirect
-	golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa // indirect
-	golang.org/x/net v0.51.0 // indirect
-	golang.org/x/sync v0.20.0 // indirect
-	golang.org/x/sys v0.41.0 // indirect
-	golang.org/x/text v0.34.0 // indirect
+	golang.org/x/crypto v0.49.0 // indirect
+	golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa // indirect
+	golang.org/x/net v0.52.0 // indirect
+	golang.org/x/sys v0.42.0 // indirect
+	golang.org/x/text v0.35.0 // indirect
 	golang.org/x/time v0.14.0 // indirect
 	golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
-	google.golang.org/genproto/googleapis/api v0.0.0-20260316172706-e463d84ca32d // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20260311181403-84a4fc48630c // indirect
-	google.golang.org/grpc v1.79.2 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	lukechampine.com/blake3 v1.4.1 // indirect