Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
618 lines (472 loc) · 21.8 KB

DEPLOYMENT.md

File metadata and controls

618 lines (472 loc) · 21.8 KB
Raw

Deployment Guide

Instructions for deploying the Stellar bulk payment system to production.

Pre-Deployment Checklist

  • All tests pass: npm test
  • Code linted: npm run lint
  • Build succeeds: npm run build
  • Testnet validation completed
  • Security review passed
  • Environment variables configured
  • Monitoring and logging setup
  • Backup and disaster recovery plan in place

Environment Setup

A complete list of supported variables with defaults and descriptions is included in .env.example. Copy it to .env and fill in your values.

cp .env.example .env

Required Environment Variables

# Production Stellar account
export STELLAR_SECRET_KEY="S..." # Never commit this!

# Optional: for enhanced security
export LOG_LEVEL="info"
export NODE_ENV="production"

ALLOW_SERVER_SIGNING G�� Server-Side Transaction Signing (#596)

Variable Default Purpose
ALLOW_SERVER_SIGNING false (unset) Allow the server to sign and submit Stellar transactions using STELLAR_SECRET_KEY

Default behaviour (unset or "false"): The API routes /api/batch-submit and /api/batch-retry reject server-signing requests with HTTP 403. Users must sign via a connected client wallet (Freighter). This is the safe default for public deployments.

When ALLOW_SERVER_SIGNING=true: The server signs transactions directly using STELLAR_SECRET_KEY. This is appropriate for:

  • Internal/trusted deployments where the server is not publicly accessible
  • Automated test pipelines (e.g. tests/batch-submit.test.ts sets this to "true")
  • Staging environments running automated batch jobs

SERVER_SIGNING_API_KEY — Cryptographic Authorization (#696, fail-closed #728)

Variable Default Purpose
SERVER_SIGNING_API_KEY "" (unset) Secret API key required in the Authorization: Bearer header for server-signing requests

When ALLOW_SERVER_SIGNING=true, the server enforces cryptographic authorization on /api/batch-submit and /api/batch-retry. Callers must include the API key in the Authorization header:

Authorization: Bearer <SERVER_SIGNING_API_KEY>

Generate a secure key:

openssl rand -hex 32
# Example output: a1b2c3d4e5f6...64-char-hex-string

Fail closed (#728): SERVER_SIGNING_API_KEY is required whenever ALLOW_SERVER_SIGNING=true. If it is not set, /api/batch-submit and /api/batch-retry refuse every server-signing request with 403 — they do not fall back to accepting requests without a credential. Server-signing moves real funds from a hot wallet, so an unconfigured credential must mean "nobody is authorized," never "everybody is authorized." Set SERVER_SIGNING_API_KEY in every deployment where ALLOW_SERVER_SIGNING=true.

SERVER_SIGNING_ALLOW_UNAUTHENTICATED — local-demo-only opt-out:

Variable Default Purpose
SERVER_SIGNING_ALLOW_UNAUTHENTICATED false (unset) Explicit, narrow opt-in to accept server-signing requests without a credential, for local demos only

If you need to exercise server-signing locally without generating a key, set SERVER_SIGNING_ALLOW_UNAUTHENTICATED=true. This is refused outright whenever the process is running in production (NODE_ENV=production or BATCHPAY_ENV=production) — the request still fails closed with 403 in that case, so this opt-in can never accidentally become a production posture. Never set this variable in a deployed environment.

Security warnings:

  • ALLOW_SERVER_SIGNING=true centralises key risk on the server. A compromised server can sign and submit arbitrary transactions.
  • SERVER_SIGNING_API_KEY is required, not optional, whenever ALLOW_SERVER_SIGNING=true — the server refuses to run server-signing requests without it (see fail-closed note above).
  • Never enable on public-facing production endpoints without additional access controls (VPN, IP allowlist, or mutual TLS).
  • Requires STELLAR_SECRET_KEY to be set; the flag has no effect without it.
  • Audit all access logs when this flag is active.
# Staging / internal use only
export ALLOW_SERVER_SIGNING=true
export STELLAR_SECRET_KEY="S..."
export SERVER_SIGNING_API_KEY="$(openssl rand -hex 32)"

# Production (public) G�� leave unset; users sign via Freighter wallet
# ALLOW_SERVER_SIGNING is intentionally absent

API error when disabled: POST /api/batch-submit or /api/batch-retry without server signing enabled returns:

{
  "error": "Server-side signing is disabled. Use client-side signing with a connected wallet, or enable ALLOW_SERVER_SIGNING=true in server configuration."
}

API error when SERVER_SIGNING_API_KEY is unset (403, fail closed):

{
  "error": "Server-signing is not authorized: SERVER_SIGNING_API_KEY is not configured on the server. Refusing this request."
}

API error for missing credential (401):

{
  "error": "Missing or malformed Authorization header. Server-signing requests require an 'Authorization: Bearer <SERVER_SIGNING_API_KEY>' header."
}

API error for invalid credential (403):

{
  "error": "Invalid server-signing API key. The provided Authorization token does not match the configured SERVER_SIGNING_API_KEY."
}

See DEVELOPMENT.md for local test setup using this flag.

WALLET_AUTH_SECRET — Wallet Session Authentication

Batch read/recover routes (/api/batch-status, /api/batch-history, /api/batch-recover, /api/batch-events) require a short-lived wallet session token issued after the connected wallet signs a SEP-10-style challenge. Knowing a public G-address alone is no longer sufficient to read another user's payroll data.

Variable Default Purpose
WALLET_AUTH_SECRET dev fallback (non-production) HMAC secret for session tokens
WALLET_AUTH_SERVER_SECRET derived from secret SEP-10 challenge server signing key
WALLET_AUTH_HOME_DOMAIN hostname from NEXT_PUBLIC_SITE_URL SEP-10 home domain
WALLET_AUTH_WEB_AUTH_DOMAIN stellar-batch-pay SEP-10 web auth domain
WALLET_AUTH_SESSION_TTL_SEC 3600 Session lifetime in seconds

Generate a production secret:

openssl rand -hex 32
export WALLET_AUTH_SECRET="<output>"

See docs/wallet-auth.md for the dashboard polling/SSE flow and API usage.

Environment Variable Management

Do NOT commit .env files or secrets to version control.

Recommended approach:

  1. Use a secret management service (AWS Secrets Manager, HashiCorp Vault, etc.)
  2. Set environment variables at deployment time
  3. Use secure environment variable providers

For Vercel deployment:

vercel env add STELLAR_SECRET_KEY

Keeper Bot Secret Management (#257)

The keeper bot (scripts/keeper.ts) reads KEEPER_SECRET from a pluggable backend configured by SECRET_BACKEND.

Backend: env (local development only)

export SECRET_BACKEND=env
export KEEPER_SECRET="S..."   # .env or shell G�� never commit
npx ts-node scripts/keeper.ts

A warning is printed at startup when using this backend in non-development environments.

Backend: aws (recommended for production)

  1. Store the keeper secret in AWS Secrets Manager: 
    aws secretsmanager create-secret \
  --name KEEPER_SECRET \
  --secret-string '{"KEEPER_SECRET":"S..."}'
  2. Attach an IAM policy granting secretsmanager:GetSecretValue to the role running the keeper bot.
  3. Set environment variables: 
    export SECRET_BACKEND=aws
export AWS_REGION=us-east-1
# AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY or use instance/task role

Backend: github (GitHub Actions CI/CD)

  1. Add KEEPER_SECRET in your repository: Settings G�� Secrets and variables G�� Actions G�� New repository secret
  2. Reference in your workflow: 
    jobs:
  keeper:
    steps:
      - name: Run keeper
        env:
          SECRET_BACKEND: github
          KEEPER_SECRET: ${{ secrets.KEEPER_SECRET }}
        run: npx ts-node scripts/keeper.ts

No secret is written to disk, logs, or intermediate environment files in the aws or github backends.

Keeper Bot Pagination Configuration (#586)

Recipients with more than MAINTENANCE_LIMIT vesting schedule entries require multiple keeper runs to receive full TTL coverage. The bot persists a per-recipient nextMaintenanceIndex cursor between runs so progress is never lost.

Variable Default Purpose
MAINTENANCE_LIMIT 10 Number of schedule indices bumped per recipient per run
KEEPER_STATE_PATH ./data/keeper-state.json JSON file storing per-recipient pagination cursors

How many runs to achieve full coverage:

If a recipient has S schedule entries and MAINTENANCE_LIMIT=L, full coverage requires ceil(S / L) consecutive keeper runs. After the final window is processed the cursor resets to 0 and the next run begins a fresh sweep.

Example: 50 schedule entries, MAINTENANCE_LIMIT=10 G�� 5 runs for full coverage.

Tuning recommendations:

  • Increase MAINTENANCE_LIMIT to cover larger recipients in fewer runs. Keep it within Soroban transaction size limits (Stellar enforces a per-transaction instruction cap; values above 50 may require fee increases or encounter simulation errors).
  • Set KEEPER_STATE_PATH to a persistent volume path in serverless/containerised deployments so the cursor survives cold starts.
  • Keeper logs per-recipient progress: watch for cursor reset to 0 lines to confirm a full sweep completed.
# Example: tune for recipients with up to 25 entries, 3 runs for full coverage
export MAINTENANCE_LIMIT=10
export KEEPER_STATE_PATH=/mnt/data/keeper-state.json

Keeper Bot Exit Codes

The keeper script (scripts/keeper.ts) uses explicit exit codes so CI workflows correctly report success or failure.

Exit Code Meaning
0 Keeper completed successfully G�� all recipients maintained, instance bumped, balance checked.
1 Keeper encountered a fatal error (missing config, RPC failure, transaction error, etc.). The alert webhook fires before the non-zero exit.

CI behaviour:

  • GitHub Actions treats exit code 0 as success (green checkmark) and any non-zero code as failure (red cross).
  • The Report failure step in .github/workflows/keeper.yml runs on if: failure() and writes a job summary with the contract ID and run metadata, then creates or comments on a GitHub issue.

Testing exit codes locally:

# Success path (valid env)
npx ts-node scripts/keeper.ts; echo "exit: $?"

# Failure path (missing CONTRACT_ID)
CONTRACT_ID= npx ts-node scripts/keeper.ts; echo "exit: $?"
# G�� prints exit: 1

The main function is exported for programmatic testing via subprocess spawn.

Smart Contract Deployment

Follow these steps to deploy and initialize the Soroban smart contract.

1. Prerequisites

Ensure you have the following installed:

1a. Fee Asset Whitelist Configuration (#543)

The batch-vesting contract enforces a single whitelisted fee asset stored in the contract Config. This prevents admin key compromise from allowing arbitrary token fee collection that could drain depositors.

Key points:

  • The fee asset is set once during contract initialization via set_config()
  • set_fee_config() no longer accepts a fee_asset parameter G�� it only sets fee_per_recipient and treasury
  • All deposit fees are automatically collected in the whitelisted asset
  • Changing the fee asset requires a full set_config() call (admin-only)

Recommended fee assets:

  • Testnet/Mainnet: Use native XLM (the network's Stellar Asset Contract address)
  • Private networks: Use the native asset SAC address for that network

Example initialization:

# 1. Deploy contract
stellar contract deploy \
  --wasm target/wasm32-unknown-unknown/release/batch_vesting.wasm \
  --source deployer \
  --network testnet

# 2. Set admin
stellar contract invoke \
  --id <CONTRACT_ID> \
  --source deployer \
  --network testnet \
  -- set_admin --admin <ADMIN_ADDRESS>

# 3. Initialize config with fee_asset (e.g., native XLM on testnet)
stellar contract invoke \
  --id <CONTRACT_ID> \
  --source deployer \
  --network testnet \
  -- set_config \
  --admin <ADMIN_ADDRESS> \
  --config '{
    "max_batch_size": 100,
    "max_schedules_per_recipient": 10,
    "upgrade_timelock": 604800,
    "fee_asset": "<XLM_SAC_ADDRESS>"
  }'

# 4. Set fee parameters (fee_asset NOT included G�� comes from config)
stellar contract invoke \
  --id <CONTRACT_ID> \
  --source deployer \
  --network testnet \
  -- set_fee_config \
  --admin <ADMIN_ADDRESS> \
  --fee_per_recipient 10000000 \
  --treasury <TREASURY_ADDRESS>

Security notes:

  • Use a Stellar multisig (M-of-N threshold) for the admin account to prevent single-key compromise
  • The fee_asset should be a liquid, trusted token (native XLM recommended)
  • Never set fee_asset to a custom/illiquid token that depositors cannot easily obtain
  • Document the chosen fee asset in your deployment runbook for transparency

2. Configure CLI Identity

Create an identity for deployment:

stellar keys generate --network testnet deployer

3. Build the Contract

Navigate to the contract directory and build the release Wasm:

cd contracts/batch-vesting
cargo build --target wasm32-unknown-unknown --release

The compiled contract will be available at: target/wasm32-unknown-unknown/release/batch_vesting.wasm

4. Deploy to Testnet

Deploy the contract and capture the Contract ID:

stellar contract deploy \
  --wasm target/wasm32-unknown-unknown/release/batch_vesting.wasm \
  --source deployer \
  --network testnet

Note

Save the returned Contract ID (starts with C...) as it is required for frontend integration.

5. Frontend Integration

Update your frontend .env file with the newly deployed Contract ID:

NEXT_PUBLIC_CONTRACT_ID="C..."

Persistence and deployment modes

The API supports two deployment modes for batch job state, idempotency, and rate limits. Set DEPLOYMENT_MODE to choose the right one.

Single-node mode (default)

DEPLOYMENT_MODE=single-node  # or omit; this is the default

Jobs and rate limits are stored in local SQLite files. This is safe when exactly one process accesses the store (e.g. next start behind a single-container deploy, PM2 in cluster mode sharing the same filesystem, or Docker with a mounted volume).

Variable Default Purpose
JOB_STORE_PATH ./data/jobs.db Durable batch job state
RATE_LIMIT_DB_PATH ./data/rate-limit.db Per-key API rate limiting
WEBHOOK_ENCRYPTION_KEY unset* Stable key for encrypting webhook secrets
WEBHOOK_ADMIN_API_KEY unset API key required for webhook management and delivery auditing

Webhook registrations are stored in the same SQLite database as jobs and delivery logs. The schema is created automatically by the job-store initialization migration, so existing databases receive the webhooks table on their next application start. Signing secrets are stored as a SHA-256 hash and authenticated ciphertext; plaintext is returned only in the create response and is never returned by the list endpoint.

WEBHOOK_ENCRYPTION_KEY must be set to the same long, random value in every process sharing JOB_STORE_PATH. If it is omitted, the application uses the configured auth secret when available, or a development fallback. Changing the key makes existing webhook secrets undecryptable.

SQLite is configured with WAL mode, busy_timeout = 5000ms, and retry-with-jitter to handle transient lock conflicts.

Warning: Setting JOB_STORE_PATH or RATE_LIMIT_DB_PATH to /tmp/* makes them ephemeral. In-flight jobs and rate-limit state are lost on restart. The health endpoint flags this.

HA mode (multi-instance)

DEPLOYMENT_MODE=ha
JOB_STORE_BACKEND=postgres   # default when ha
RATE_LIMIT_BACKEND=redis     # default when ha (also supports postgres)
DATABASE_URL=postgres://user:pass@host/dbname
REDIS_URL=redis://host:6379

Jobs and idempotency keys are stored in Postgres; rate limits use Redis (or Postgres). All replicas share the same state, so:

  • Idempotent submit is globally convergent (no duplicate payments).
  • Rate limits apply fleet-wide.
  • In-flight jobs survive cold starts and replica replacement.
Variable Required when Purpose
DATABASE_URL JOB_STORE_BACKEND=postgres Postgres connection string
REDIS_URL RATE_LIMIT_BACKEND=redis Redis connection string
JOB_STORE_BACKEND Optional; defaults to postgres sqlite or postgres
RATE_LIMIT_BACKEND Optional; defaults to redis sqlite, postgres, redis

The health endpoint (GET /api/health) reports deploymentMode, each backend's connectivity status, and returns 503 when any store is unreachable or misconfigured. Use it as a readiness probe.

Which mode is safe?

Topology Mode Safe?
Single container / single process single-node G��
Multiple replicas, shared volume single-node G��n+� Only with a single writer
Multiple replicas, no shared disk ha G��
Serverless (ephemeral /tmp) ha G��
Serverless (ephemeral /tmp) single-node G�� Data lost on cold start

Health check G�� verify store connectivity before routing traffic:

curl -s http://localhost:3000/api/health
# Returns 200 with backend info, or 503 with config issues / connectivity errors

Set persistence variables in the same environment as your API routes (Vercel project settings, Docker env, or systemd unit).

Hosting Options

Option 1: Vercel (Recommended for Next.js)

Vercel is optimized for Next.js applications:

# Install Vercel CLI
npm install -g vercel

# Deploy
vercel

# Configure environment variables
vercel env add STELLAR_SECRET_KEY

# View deployment
vercel --prod

Advantages:

  • Zero-config deployment
  • Automatic scaling
  • Global CDN
  • Preview deployments
  • Easy rollback

Option 2: Docker Container

For flexibility and multi-platform deployment, use the committed Dockerfile at the repo root. It is a multi-stage build that:

  1. Builder stage (node:22-alpine): Installs build dependencies for native modules (Python, make, g++) and runs npm ci + npm run build
  2. Production stage (node:22-alpine): Copies only production artifacts, creates a non-root nodejs user, and includes a healthcheck
# Builder stage
FROM node:22-alpine AS builder
RUN apk add --no-cache python3 make g++ libc-dev
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Production stage
FROM node:22-alpine AS production
RUN addgroup -S nodejs && adduser -S nodejs -G nodejs
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public
COPY --from=builder /app/lib ./lib
COPY --from=builder /app/scripts ./scripts
RUN mkdir -p /app/data && chown -R nodejs:nodejs /app
USER nodejs
ENV NODE_ENV=production
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1
CMD ["node", "server.js"]

The accompanying .dockerignore keeps node_modules, .next, contracts/target, .env*, and data/ out of the build context.

Build:

docker build -t stellar-bulk-pay:latest .

Run locally:

docker run --rm -p 3000:3000 \
  -e NODE_ENV=production \
  -e STELLAR_SECRET_KEY="$STELLAR_SECRET_KEY" \
  -v "$(pwd)/data:/app/data" \
  stellar-bulk-pay:latest

Push:

docker tag stellar-bulk-pay:latest myregistry/stellar-bulk-pay:latest
docker push myregistry/stellar-bulk-pay:latest

Deploy to container service:

  • AWS ECS — mount an EFS volume at /app/data if you need durable SQLite.
  • Google Cloud Run — pair with a managed database, or accept that data/ resets on each container instance.
  • Azure Container Instances

Option 3: Traditional VPS