feat: one-command local dev stack (make localnet) with seed, monitoring, and a fresh-init migration fix - #236
Merged
Conversation
Adds deploy/localnet: docker-compose (API + TimescaleDB Postgres + OTel collector + Prometheus + Grafana preloaded with the repo's dashboards), Makefile targets (localnet / localnet-seed / localnet-mock / localnet-logs / localnet-clean), a seed script provisioning 8 validators with stakes, reputations and pending rewards, and a mock telemetry generator emitting uptime_heartbeat + reward_tx windows. Scope confirmed with maintainer on the issue: genesis/pre-funded accounts and mock attestations are mapped to their real equivalents in this backend's schema. Also fixes a pre-existing migration bug surfaced by live-testing the stack: 011_distributed_jobs.sql used NOW() in a partial-index predicate, which Postgres rejects (functions in index predicates must be IMMUTABLE) — this aborted every fresh DB init, including the app's own migration_manager path. The index keeps its static status filter; the lock-expiry condition stays in queries, which can still use the index. Verified live: fresh container applies all 14 migrations with 0 errors (timescaledb + pg_cron active), seed + mock scripts run green (8 validators, 248 heartbeats, 24 reward_tx confirmed in the DB). Closes VeriNode-Labs#210
github-actions Bot
pushed a commit
to starfishkrq/VeriNode-Backend
that referenced
this pull request
Aug 24, 2026
…version Float Variations VeriNode-Labs#217 (#1) * Add auto-assign workflow * feat(mtls): service mesh mTLS inter-service authentication VeriNode-Labs#202 - Add parseVeriNodeSpiffeId/buildVeriNodeSpiffeId for spiffe://verinode.labs/{service_name}/{pod_id} identity format - Add verifyPeerServiceIdentity and extractVeriNodeServiceName for incoming connection validation - Add mTLS handshake latency histogram (verinode_mtls_handshake_duration_ms) with recordHandshakeLatency method - Expose HandshakeLatencyBuckets in MtlsMetricsSnapshot and Prometheus text output - Add integration test (tests/security/mtls_integration.test.ts): 3 services with distinct verinode.labs SPIFFE IDs, authorized cross-call success, unauthorized rejection (403), latency histogram observations, cert rotation hot-reload detection - Update deploy/mtls/cert-manager.yaml: CA CN changed to verinode.labs; workload cert adds spiffe://verinode.labs/verinode-backend/default URI SAN alongside cluster.local fallback - Update deploy/mtls/istio-mtls.yaml: AuthorizationPolicy extended with requestPrincipals for spiffe://verinode.labs/* prefix - Update deploy/mtls/monitoring.yaml: Add VeriNodeMtlsHandshakeLatencyHigh alert (P99 > 100ms) - Add ci:validate-workflow script to package.json for CI shard compliance - Register mtls_integration.test.ts in scripts/run-tests.cjs * Fix auto-assign workflow * Add auto-merge workflow for PRs * fix: Robustness improvements for Two-Phase Commit Controller * fix: Resolve CI build failure (redis version, strict TS, pinned deps) * chore: Update pnpm-lock.yaml to match strictly pinned dependencies * chore: Align CI with npm, fix security gate and coverage scripts * fix: Remove orphaned with: blocks in ci.yml * fix: Update test runner, install vitest, fix failing tests, and update CI validator * fix: Restore missing ci:validate-workflow script and lower coverage thresholds * chore: remove node_modules from version control * fix: replace ts-node with tsx to resolve c8 module not found error in CI * fix: resolve failing mtls_integration and payload_encryption tests under new test runner * feat: Automated Certificate Lifecycle with ACME Protocol and Zero-Downtime Reload VeriNode-Labs#206 (VeriNode-Labs#228) * feat: Automated Certificate Lifecycle with ACME Protocol and Zero-Downtime Reload VeriNode-Labs#206 - Add AcmeDns01Issuer: DNS-01 challenge solver using a pluggable Dns01ChallengeStore interface (setTxtRecord/removeTxtRecord), enabling wildcard and DNS-based issuance alongside the existing HTTP-01 AcmeClientIssuer. - Add CertLifecycleMetrics: per-service cert_expiry_days Prometheus gauge, renewal attempt/success/failure counters; alerts when days_remaining < 14 (CERT_ALERT_DAYS). - Add CertLifecycleManager: multi-service certificate lifecycle orchestrator that stores certs at /etc/verinode/certs/{service}/{cert,key,chain}.pem; daily cron check (configurable checkIntervalMs) across all registered services; falls back to existing cert on renewal failure until < 7-day emergency window; emits structured WARN logs and onAlert callbacks when cert < 14 days from expiry; exposes checkAllOnce(), checkServiceOnce(), getAllStatus(), getServiceStatus(), prometheusMetrics() for operational introspection. - Add registerCertManagementRoutes: mounts POST /api/v1/certs/renew and GET /api/v1/certs/status on an Express app instance; renew accepts optional { service } body to target a single service or all services when omitted. - Add tests/tls_rotation.test.ts: 37 tests covering CertificateStore, FileChallengeStore, AcmeDns01Issuer, CertLifecycleMetrics, AcmeRenewalManager (including integration test: issue cert -> fast-forward 25 days -> verify auto-renewal triggers and cert replaced), CertLifecycleManager, registerCertManagementRoutes, and TlsCertificateReloader. All tests pass (37/37). Compatible with scripts/run-tests.cjs test runner. * fix: sync lockfiles and correct redis version for CI VeriNode-Labs#206 - Fix redis version from ^4.9.0 (non-existent) to ^4.7.0 (latest stable) - Regenerate package-lock.json to include redis and all missing deps - Regenerate pnpm-lock.yaml to include redis, acme-client, @types/express, @types/json-schema, and c8 which were missing from the lockfile - Fix pnpm-workspace.yaml: replace invalid placeholder config with correct pnpm v9 packages/onlyBuiltDependencies format so pnpm install works * feat: add Prometheus metrics scrape target and perf regression detection (VeriNode-Labs#225) Closes VeriNode-Labs#214, closes VeriNode-Labs#207, closes VeriNode-Labs#208 ## Issue VeriNode-Labs#214 — Prometheus Scrape Target: Thread State Metrics Implements src/api/metrics/ module exposing: - Thread state metrics from /proc/self/task/*/status (Linux), with graceful fallback on non-Linux hosts. Labels: thread_name, thread_state (Running/Sleeping/Blocked/Deadlocked/Zombie) - Node.js runtime metrics mapped to tokio-equivalent names: verinode_worker_poll_duration_seconds, verinode_worker_queue_depth, verinode_num_alive_tasks, verinode_num_blocking_threads, verinode_io_driver_ready_count - Connection pool gauges: verinode_pool_connections_active/idle{pool=oltp|olap} - Ledger confirmation lag: verinode_ledger_confirmation_lag_seconds - HTTP request duration histogram with [0.001..5.0] s buckets, per (route, method, status_code) labels - OpenTelemetry exemplar support: attaches trace_id to histogram observations when a trace span is active - GET /debug/metrics/check self-test endpoint (PASS/FAIL JSON) - MetricsRegistry for collector registration No external prometheus client library required — follows the same manual Prometheus text-format rendering pattern used in scheduler/metrics.ts. ## Issues VeriNode-Labs#207 / VeriNode-Labs#208 — Automated Performance Regression Detection Implements src/performance/ module with: - EDM (E-Divisive with Medians) change point detection algorithm - BenchmarkRunner: runs scenarios for fixed duration, computes p50/p95/p99 latency, throughput (rps), and error rate - BaselineStore: persists baselines as JSON files keyed by {branch}_{scenario}.json (compatible with S3-mounted paths) - RegressionDetector: compares metrics against baseline, flags any metric with >2% degradation; generates markdown PR comment table - Full documentation in docs/performance-regression.md * feat: modular pre-commit hook suite with per-hook skip and CI parity Add pre-commit.d/ hook scripts for format, lint, debug, secrets, and large file checks. Orchestrator enforces 30s timeout with SKIP=hook-name bypass. CI runs same checks. Closes VeriNode-Labs#203. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com> * chore: remove unused API routes and middleware (VeriNode-Labs#231) Co-authored-by: zinodict121 <dev373055@gmail.com> * Fix all failing unit and integration tests (VeriNode-Labs#232) Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com> * chore: add eslint and prettier configuration and auto-fix style issues (VeriNode-Labs#233) Co-authored-by: zinodict121 <dev373055@gmail.com> Co-authored-by: JamesEjembi <jameseonoja@gmail.com> * docs: consolidate all documentation into single backend guide (VeriNode-Labs#234) Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com> * chore: remove unused backend dependencies (VeriNode-Labs#227) Co-authored-by: JamesEjembi <jameseonoja@gmail.com> * feat: add validator key rotation ceremony (VeriNode-Labs#230) * fix: make slashing penalty calculation race-safe (VeriNode-Labs#235) * chore(deps): bump actions/checkout from 4 to 7 (VeriNode-Labs#237) Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: one-command local dev stack (make localnet) with seed + monitoring (VeriNode-Labs#236) Adds deploy/localnet: docker-compose (API + TimescaleDB Postgres + OTel collector + Prometheus + Grafana preloaded with the repo's dashboards), Makefile targets (localnet / localnet-seed / localnet-mock / localnet-logs / localnet-clean), a seed script provisioning 8 validators with stakes, reputations and pending rewards, and a mock telemetry generator emitting uptime_heartbeat + reward_tx windows. Scope confirmed with maintainer on the issue: genesis/pre-funded accounts and mock attestations are mapped to their real equivalents in this backend's schema. Also fixes a pre-existing migration bug surfaced by live-testing the stack: 011_distributed_jobs.sql used NOW() in a partial-index predicate, which Postgres rejects (functions in index predicates must be IMMUTABLE) — this aborted every fresh DB init, including the app's own migration_manager path. The index keeps its static status filter; the lock-expiry condition stays in queries, which can still use the index. Verified live: fresh container applies all 14 migrations with 0 errors (timescaledb + pg_cron active), seed + mock scripts run green (8 validators, 248 heartbeats, 24 reward_tx confirmed in the DB). Closes VeriNode-Labs#210 * feat(math): fixed-point arithmetic safeguards for staking reward conversions (VeriNode-Labs#217) Closes VeriNode-Labs#217 What was changed: - src/utils/math_precision.ts: New FixedPoint struct backed by bigint (i128 semantics). Exposes add, sub, mul, div (all floor-integer, zero IEEE 754), fromRatio() for precision-safe ratio construction, abs(), toString() (7-dec string), toUnits(). Includes assertDistributionSumcheck() that panics with an invariant violation if |sum(distributed) - pool| > 1 unit across a cycle. - src/utils/math_float.ts: Deprecated legacy IEEE 754 float path retained for migration window only. JSDoc @deprecated annotation flags all callers in IDEs. - src/rewards/compute_engine.ts: New fixed-point reward engine. calculateReward() replaces float chain with integer-domain computation: composite = fromRatio(uptime) x fromRatio(compute/1M) x fromRatio(storage/1M) units = composite.raw x (totalPoolUnits / nodeCount) / SCALE calculateBatchRewards() adds per-cycle sumcheck assertion. - src/staking/weight_aggregator.ts: aggregateWeights() computes per-node composite weights and normalised pool fractions using FixedPoint integer ops. - tests/rewards/distributor.test.ts: 28 deterministic tests including 1000- iteration property-based comparison: |fixed - float| < 1 unit across random weight triplets, sumcheck pass/fail boundaries, 50k-node large-pool scenario. - CHANGELOG.md: Migration guide for downstream callers of math_float.ts. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: JamesEjembi <jameseonoja@gmail.com> Co-authored-by: mona-i <josephenemona300@gmail.com> Co-authored-by: Raj <rasalraj10@gmail.com> Co-authored-by: Raj Rasal <76217705+Soldier224K@users.noreply.github.com> Co-authored-by: Joey <62303285+Mona-i@users.noreply.github.com> Co-authored-by: Husten150 <160516146+Husten150@users.noreply.github.com> Co-authored-by: Okorie Chigozie Jehoshaphat <okoriechigozie99@gmail.com> Co-authored-by: Codebuff <noreply@codebuff.com> Co-authored-by: zinodict121 <b02579299@gmail.com> Co-authored-by: zinodict121 <dev373055@gmail.com> Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com> Co-authored-by: telemarkdigital-publisher <thetelemarkdigital@gmail.com> Co-authored-by: Dylan Rojas <dyrojasar@est.utn.ac.cr> Co-authored-by: JerryIdoko <Onojajerome04@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bernice <100929843+addnad@users.noreply.github.com> Co-authored-by: VeriNode Dev <dev@verinode.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One-command local development environment, per the scope confirmed on the issue (#210 comment):
make localnetboots the API + TimescaleDB Postgres (migrations auto-applied) + OTel collector + Prometheus + Grafana (pre-provisioned with the repo's existing dashboards), then seeds validators and mock telemetry.make localnet-cleanremoves all state.What's included
deploy/localnet/docker-compose.yml— postgres (timescale/timescaledb-ha:pg16, withpg_cronpreloaded so migration 002 can install it), otel-collector (OTLP gRPC :4317), api (built from the repo Dockerfile), prometheus (scrapes the API's/metrics), grafana (:3001, anonymous viewer, Prometheus datasource + the dashboards fromdeploy/observability/anddeploy/monitoring/file-provisioned).scripts/seed-localnet.ts— idempotent seed: a bond pool + 8 validators, each with a stake (validator_stakes), reputation (reputations), and pending rewards (reward_pending_amounts) — the "pre-funded accounts" equivalent in this schema.scripts/mock-telemetry.ts— emits a window ofuptime_heartbeatrows +reward_txper validator — the "mock attestation" equivalent.Makefile—localnet,localnet-seed,localnet-mock,localnet-logs,localnet-clean. Host port for Postgres is overridable (LOCALNET_PG_PORT) if 5432 is taken.deploy/localnet/README.md— usage + the scope mapping.Pre-existing migration bug found & fixed (blocks every fresh DB init)
Live-testing the stack surfaced that
011_distributed_jobs.sqlfails on any fresh database: its partial index usedNOW()in the predicate —Postgres rejects this (
functions in index predicate must be marked IMMUTABLE) because index membership is fixed at write time whileNOW()changes continuously — so the migration aborts under initdb and under the app's ownmigration_manager(same SQL). The fix keeps the staticstatusfilter (which excludes the bulk of the table) and leaves the time-dependentlocked_untilcheck at query time, where lease queries already apply it and can still use the index.Verification (all run live)
timescaledb+pg_cronextensions active; the fixedidx_distributed_jobs_readyexists.seed-localnet→ 8 validators seeded (verifiedvalidator_stakescount = 8).mock-telemetry→ 248 heartbeat rows + 24 reward_tx confirmed in the DB.docker compose build api(repo Dockerfile:npm ci+tsc) → builds clean.docker compose configvalidates.Closes #210