Skip to content

feat: one-command local dev stack (make localnet) with seed, monitoring, and a fresh-init migration fix - #236

Merged
JamesEjembi merged 1 commit into
VeriNode-Labs:mainfrom
addnad:feat/localnet-setup
Aug 24, 2026
Merged

feat: one-command local dev stack (make localnet) with seed, monitoring, and a fresh-init migration fix#236
JamesEjembi merged 1 commit into
VeriNode-Labs:mainfrom
addnad:feat/localnet-setup

Conversation

@addnad

@addnad addnad commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Summary

One-command local development environment, per the scope confirmed on the issue (#210 comment): make localnet boots the API + TimescaleDB Postgres (migrations auto-applied) + OTel collector + Prometheus + Grafana (pre-provisioned with the repo's existing dashboards), then seeds validators and mock telemetry. make localnet-clean removes all state.

make localnet         # build + boot + seed
make localnet-logs    # tail services
make localnet-clean   # tear down, remove volumes

What's included

  • deploy/localnet/docker-compose.yml — postgres (timescale/timescaledb-ha:pg16, with pg_cron preloaded so migration 002 can install it), otel-collector (OTLP gRPC :4317), api (built from the repo Dockerfile), prometheus (scrapes the API's /metrics), grafana (:3001, anonymous viewer, Prometheus datasource + the dashboards from deploy/observability/ and deploy/monitoring/ file-provisioned).
  • scripts/seed-localnet.ts — idempotent seed: a bond pool + 8 validators, each with a stake (validator_stakes), reputation (reputations), and pending rewards (reward_pending_amounts) — the "pre-funded accounts" equivalent in this schema.
  • scripts/mock-telemetry.ts — emits a window of uptime_heartbeat rows + reward_tx per validator — the "mock attestation" equivalent.
  • Makefilelocalnet, localnet-seed, localnet-mock, localnet-logs, localnet-clean. Host port for Postgres is overridable (LOCALNET_PG_PORT) if 5432 is taken.
  • deploy/localnet/README.md — usage + the scope mapping.

Pre-existing migration bug found & fixed (blocks every fresh DB init)

Live-testing the stack surfaced that 011_distributed_jobs.sql fails on any fresh database: its partial index used NOW() in the predicate —

WHERE status IN ('pending','running') AND (locked_until IS NULL OR locked_until <= NOW());

Postgres rejects this (functions in index predicate must be marked IMMUTABLE) because index membership is fixed at write time while NOW() changes continuously — so the migration aborts under initdb and under the app's own migration_manager (same SQL). The fix keeps the static status filter (which excludes the bulk of the table) and leaves the time-dependent locked_until check at query time, where lease queries already apply it and can still use the index.

Verification (all run live)

  • Fresh container init applies all 14 migrations with 0 errors; timescaledb + pg_cron extensions active; the fixed idx_distributed_jobs_ready exists.
  • seed-localnet → 8 validators seeded (verified validator_stakes count = 8).
  • mock-telemetry248 heartbeat rows + 24 reward_tx confirmed in the DB.
  • docker compose build api (repo Dockerfile: npm ci + tsc) → builds clean.
  • docker compose config validates.

Closes #210

Adds deploy/localnet: docker-compose (API + TimescaleDB Postgres + OTel
collector + Prometheus + Grafana preloaded with the repo's dashboards),
Makefile targets (localnet / localnet-seed / localnet-mock / localnet-logs /
localnet-clean), a seed script provisioning 8 validators with stakes,
reputations and pending rewards, and a mock telemetry generator emitting
uptime_heartbeat + reward_tx windows. Scope confirmed with maintainer on
the issue: genesis/pre-funded accounts and mock attestations are mapped to
their real equivalents in this backend's schema.

Also fixes a pre-existing migration bug surfaced by live-testing the stack:
011_distributed_jobs.sql used NOW() in a partial-index predicate, which
Postgres rejects (functions in index predicates must be IMMUTABLE) — this
aborted every fresh DB init, including the app's own migration_manager path.
The index keeps its static status filter; the lock-expiry condition stays in
queries, which can still use the index.

Verified live: fresh container applies all 14 migrations with 0 errors
(timescaledb + pg_cron active), seed + mock scripts run green (8 validators,
248 heartbeats, 24 reward_tx confirmed in the DB).

Closes VeriNode-Labs#210
@JamesEjembi
JamesEjembi merged commit dcd5e5f into VeriNode-Labs:main Aug 24, 2026
10 of 16 checks passed
github-actions Bot pushed a commit to starfishkrq/VeriNode-Backend that referenced this pull request Aug 24, 2026
…version Float Variations VeriNode-Labs#217 (#1)

* Add auto-assign workflow

* feat(mtls): service mesh mTLS inter-service authentication VeriNode-Labs#202

- Add parseVeriNodeSpiffeId/buildVeriNodeSpiffeId for spiffe://verinode.labs/{service_name}/{pod_id} identity format
- Add verifyPeerServiceIdentity and extractVeriNodeServiceName for incoming connection validation
- Add mTLS handshake latency histogram (verinode_mtls_handshake_duration_ms) with recordHandshakeLatency method
- Expose HandshakeLatencyBuckets in MtlsMetricsSnapshot and Prometheus text output
- Add integration test (tests/security/mtls_integration.test.ts): 3 services with distinct verinode.labs SPIFFE IDs, authorized cross-call success, unauthorized rejection (403), latency histogram observations, cert rotation hot-reload detection
- Update deploy/mtls/cert-manager.yaml: CA CN changed to verinode.labs; workload cert adds spiffe://verinode.labs/verinode-backend/default URI SAN alongside cluster.local fallback
- Update deploy/mtls/istio-mtls.yaml: AuthorizationPolicy extended with requestPrincipals for spiffe://verinode.labs/* prefix
- Update deploy/mtls/monitoring.yaml: Add VeriNodeMtlsHandshakeLatencyHigh alert (P99 > 100ms)
- Add ci:validate-workflow script to package.json for CI shard compliance
- Register mtls_integration.test.ts in scripts/run-tests.cjs

* Fix auto-assign workflow

* Add auto-merge workflow for PRs

* fix: Robustness improvements for Two-Phase Commit Controller

* fix: Resolve CI build failure (redis version, strict TS, pinned deps)

* chore: Update pnpm-lock.yaml to match strictly pinned dependencies

* chore: Align CI with npm, fix security gate and coverage scripts

* fix: Remove orphaned with: blocks in ci.yml

* fix: Update test runner, install vitest, fix failing tests, and update CI validator

* fix: Restore missing ci:validate-workflow script and lower coverage thresholds

* chore: remove node_modules from version control

* fix: replace ts-node with tsx to resolve c8 module not found error in CI

* fix: resolve failing mtls_integration and payload_encryption tests under new test runner

* feat: Automated Certificate Lifecycle with ACME Protocol and Zero-Downtime Reload VeriNode-Labs#206 (VeriNode-Labs#228)

* feat: Automated Certificate Lifecycle with ACME Protocol and Zero-Downtime Reload VeriNode-Labs#206

- Add AcmeDns01Issuer: DNS-01 challenge solver using a pluggable Dns01ChallengeStore
  interface (setTxtRecord/removeTxtRecord), enabling wildcard and DNS-based issuance
  alongside the existing HTTP-01 AcmeClientIssuer.

- Add CertLifecycleMetrics: per-service cert_expiry_days Prometheus gauge, renewal
  attempt/success/failure counters; alerts when days_remaining < 14 (CERT_ALERT_DAYS).

- Add CertLifecycleManager: multi-service certificate lifecycle orchestrator that
  stores certs at /etc/verinode/certs/{service}/{cert,key,chain}.pem; daily cron
  check (configurable checkIntervalMs) across all registered services; falls back to
  existing cert on renewal failure until < 7-day emergency window; emits structured
  WARN logs and onAlert callbacks when cert < 14 days from expiry; exposes
  checkAllOnce(), checkServiceOnce(), getAllStatus(), getServiceStatus(),
  prometheusMetrics() for operational introspection.

- Add registerCertManagementRoutes: mounts POST /api/v1/certs/renew and
  GET /api/v1/certs/status on an Express app instance; renew accepts optional
  { service } body to target a single service or all services when omitted.

- Add tests/tls_rotation.test.ts: 37 tests covering CertificateStore, FileChallengeStore,
  AcmeDns01Issuer, CertLifecycleMetrics, AcmeRenewalManager (including integration test:
  issue cert -> fast-forward 25 days -> verify auto-renewal triggers and cert replaced),
  CertLifecycleManager, registerCertManagementRoutes, and TlsCertificateReloader.
  All tests pass (37/37). Compatible with scripts/run-tests.cjs test runner.

* fix: sync lockfiles and correct redis version for CI VeriNode-Labs#206

- Fix redis version from ^4.9.0 (non-existent) to ^4.7.0 (latest stable)
- Regenerate package-lock.json to include redis and all missing deps
- Regenerate pnpm-lock.yaml to include redis, acme-client, @types/express,
  @types/json-schema, and c8 which were missing from the lockfile
- Fix pnpm-workspace.yaml: replace invalid placeholder config with correct
  pnpm v9 packages/onlyBuiltDependencies format so pnpm install works

* feat: add Prometheus metrics scrape target and perf regression detection (VeriNode-Labs#225)

Closes VeriNode-Labs#214, closes VeriNode-Labs#207, closes VeriNode-Labs#208

## Issue VeriNode-Labs#214 — Prometheus Scrape Target: Thread State Metrics

Implements src/api/metrics/ module exposing:
- Thread state metrics from /proc/self/task/*/status (Linux),
  with graceful fallback on non-Linux hosts.
  Labels: thread_name, thread_state (Running/Sleeping/Blocked/Deadlocked/Zombie)
- Node.js runtime metrics mapped to tokio-equivalent names:
  verinode_worker_poll_duration_seconds, verinode_worker_queue_depth,
  verinode_num_alive_tasks, verinode_num_blocking_threads,
  verinode_io_driver_ready_count
- Connection pool gauges: verinode_pool_connections_active/idle{pool=oltp|olap}
- Ledger confirmation lag: verinode_ledger_confirmation_lag_seconds
- HTTP request duration histogram with [0.001..5.0] s buckets,
  per (route, method, status_code) labels
- OpenTelemetry exemplar support: attaches trace_id to histogram
  observations when a trace span is active
- GET /debug/metrics/check self-test endpoint (PASS/FAIL JSON)
- MetricsRegistry for collector registration

No external prometheus client library required — follows the same
manual Prometheus text-format rendering pattern used in scheduler/metrics.ts.

## Issues VeriNode-Labs#207 / VeriNode-Labs#208 — Automated Performance Regression Detection

Implements src/performance/ module with:
- EDM (E-Divisive with Medians) change point detection algorithm
- BenchmarkRunner: runs scenarios for fixed duration, computes
  p50/p95/p99 latency, throughput (rps), and error rate
- BaselineStore: persists baselines as JSON files keyed by
  {branch}_{scenario}.json (compatible with S3-mounted paths)
- RegressionDetector: compares metrics against baseline, flags any
  metric with >2% degradation; generates markdown PR comment table
- Full documentation in docs/performance-regression.md

* feat: modular pre-commit hook suite with per-hook skip and CI parity

Add pre-commit.d/ hook scripts for format, lint, debug, secrets, and
large file checks. Orchestrator enforces 30s timeout with SKIP=hook-name
bypass. CI runs same checks. Closes VeriNode-Labs#203.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>

* chore: remove unused API routes and middleware (VeriNode-Labs#231)

Co-authored-by: zinodict121 <dev373055@gmail.com>

* Fix all failing unit and integration tests (VeriNode-Labs#232)

Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com>

* chore: add eslint and prettier configuration and auto-fix style issues (VeriNode-Labs#233)

Co-authored-by: zinodict121 <dev373055@gmail.com>
Co-authored-by: JamesEjembi <jameseonoja@gmail.com>

* docs: consolidate all documentation into single backend guide (VeriNode-Labs#234)

Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com>

* chore: remove unused backend dependencies (VeriNode-Labs#227)

Co-authored-by: JamesEjembi <jameseonoja@gmail.com>

* feat: add validator key rotation ceremony (VeriNode-Labs#230)

* fix: make slashing penalty calculation race-safe (VeriNode-Labs#235)

* chore(deps): bump actions/checkout from 4 to 7 (VeriNode-Labs#237)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: one-command local dev stack (make localnet) with seed + monitoring (VeriNode-Labs#236)

Adds deploy/localnet: docker-compose (API + TimescaleDB Postgres + OTel
collector + Prometheus + Grafana preloaded with the repo's dashboards),
Makefile targets (localnet / localnet-seed / localnet-mock / localnet-logs /
localnet-clean), a seed script provisioning 8 validators with stakes,
reputations and pending rewards, and a mock telemetry generator emitting
uptime_heartbeat + reward_tx windows. Scope confirmed with maintainer on
the issue: genesis/pre-funded accounts and mock attestations are mapped to
their real equivalents in this backend's schema.

Also fixes a pre-existing migration bug surfaced by live-testing the stack:
011_distributed_jobs.sql used NOW() in a partial-index predicate, which
Postgres rejects (functions in index predicates must be IMMUTABLE) — this
aborted every fresh DB init, including the app's own migration_manager path.
The index keeps its static status filter; the lock-expiry condition stays in
queries, which can still use the index.

Verified live: fresh container applies all 14 migrations with 0 errors
(timescaledb + pg_cron active), seed + mock scripts run green (8 validators,
248 heartbeats, 24 reward_tx confirmed in the DB).

Closes VeriNode-Labs#210

* feat(math): fixed-point arithmetic safeguards for staking reward conversions (VeriNode-Labs#217)

Closes VeriNode-Labs#217

What was changed:
- src/utils/math_precision.ts: New FixedPoint struct backed by bigint (i128
  semantics). Exposes add, sub, mul, div (all floor-integer, zero IEEE 754),
  fromRatio() for precision-safe ratio construction, abs(), toString() (7-dec
  string), toUnits(). Includes assertDistributionSumcheck() that panics with an
  invariant violation if |sum(distributed) - pool| > 1 unit across a cycle.

- src/utils/math_float.ts: Deprecated legacy IEEE 754 float path retained for
  migration window only. JSDoc @deprecated annotation flags all callers in IDEs.

- src/rewards/compute_engine.ts: New fixed-point reward engine. calculateReward()
  replaces float chain with integer-domain computation:
    composite = fromRatio(uptime) x fromRatio(compute/1M) x fromRatio(storage/1M)
    units     = composite.raw x (totalPoolUnits / nodeCount) / SCALE
  calculateBatchRewards() adds per-cycle sumcheck assertion.

- src/staking/weight_aggregator.ts: aggregateWeights() computes per-node
  composite weights and normalised pool fractions using FixedPoint integer ops.

- tests/rewards/distributor.test.ts: 28 deterministic tests including 1000-
  iteration property-based comparison: |fixed - float| < 1 unit across random
  weight triplets, sumcheck pass/fail boundaries, 50k-node large-pool scenario.

- CHANGELOG.md: Migration guide for downstream callers of math_float.ts.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: JamesEjembi <jameseonoja@gmail.com>
Co-authored-by: mona-i <josephenemona300@gmail.com>
Co-authored-by: Raj <rasalraj10@gmail.com>
Co-authored-by: Raj Rasal <76217705+Soldier224K@users.noreply.github.com>
Co-authored-by: Joey <62303285+Mona-i@users.noreply.github.com>
Co-authored-by: Husten150 <160516146+Husten150@users.noreply.github.com>
Co-authored-by: Okorie Chigozie Jehoshaphat <okoriechigozie99@gmail.com>
Co-authored-by: Codebuff <noreply@codebuff.com>
Co-authored-by: zinodict121 <b02579299@gmail.com>
Co-authored-by: zinodict121 <dev373055@gmail.com>
Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com>
Co-authored-by: telemarkdigital-publisher <thetelemarkdigital@gmail.com>
Co-authored-by: Dylan Rojas <dyrojasar@est.utn.ac.cr>
Co-authored-by: JerryIdoko <Onojajerome04@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bernice <100929843+addnad@users.noreply.github.com>
Co-authored-by: VeriNode Dev <dev@verinode.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Developer Onboarding Script for Local Validator Testnet Setup

2 participants