Skip to content

fix: make slashing penalty calculation race-safe - #235

Merged
JamesEjembi merged 1 commit into
VeriNode-Labs:mainfrom
drojas1316:feature/slashing-penalty-race-212
Aug 23, 2026
Merged

fix: make slashing penalty calculation race-safe#235
JamesEjembi merged 1 commit into
VeriNode-Labs:mainfrom
drojas1316:feature/slashing-penalty-race-212

Conversation

@drojas1316

Copy link
Copy Markdown
Contributor

Closes #212

Summary

Fixes the slashing penalty race described in #212 by ensuring the active validator count and slashing event are handled inside the same PostgreSQL transaction.

The implementation serializes validator-set changes against penalty calculation, persists the validator count used at the exact slashing point, and adds deterministic concurrency coverage for joins, activation, deactivation, and concurrent slashing operations.

Related issue

Closes #212

What changed

  • Added validator_registry as the authoritative validator-set membership table.
  • Added active_validators as a view over active registry entries.
  • Added slashing_events with persisted validator_count_at_slashing.
  • Added ValidatorRegistry for validator registration, activation, deactivation, and active-set queries.
  • Added monetary calculatePenalty() logic using the required base penalty of 500 tokens.
  • Added SlashingExecutor to calculate and persist penalties atomically.
  • Added migration 012_slashing_penalty_consistency.sql.
  • Added deterministic concurrent race tests.

Transaction consistency

Each slashing event uses a single PostgreSQL client and follows this exact order:

BEGIN ISOLATION LEVEL SERIALIZABLE
LOCK TABLE validator_registry IN SHARE ROW EXCLUSIVE MODE
SELECT COUNT(*) AS count FROM active_validators
calculate penalty
INSERT INTO slashing_events (..., validator_count_at_slashing)
COMMIT

On failure, the transaction is rolled back and the client is always released.

This prevents validator joins or leaves from becoming visible between the active-set snapshot and the persisted slashing event.

Race behavior

If slashing acquires the registry lock first:

  1. The active validator count is read.
  2. Concurrent validator-set changes wait.
  3. The slashing event is persisted with that historical count.
  4. Slashing commits.
  5. The membership change proceeds.

If the membership update commits first, the following slashing transaction sees the new count.

Both outcomes correspond to a valid serial execution.

Penalty calculation

The protocol base penalty is fixed at 500 tokens.

The multiplier is:

1 + (totalValidators - activeValidators) / totalValidators
Active / Total Multiplier Penalty
10 / 10 1.00 500
3 / 4 1.25 625
0 / 10 2.00 1000

Invalid validator counts are rejected instead of silently clamped.

Concurrent coverage

The focused tests cover:

  • a new validator being registered while slashing holds the registry lock
  • activation during slashing
  • deactivation during slashing
  • membership mutation completing before slashing
  • two concurrent slashing transactions
  • historical snapshot stability after membership changes
  • exact transaction query ordering
  • rollback behavior on count, calculation, insert, and commit failures
  • client release on success and failure paths
  • formula boundaries and invalid inputs

The concurrent registration test verifies that the new validator is not visible until slashing commits, while the stored event retains the original validatorCountAtSlashing.

Validation

Focused race test:

npx tsx tests/slashing/penalty_calculator_race.test.ts
slashing penalty consistency tests passed

Focused strict TypeScript compilation:

PASS — no diagnostics

Existing migration regression:

npx tsx tests/database/migration_manager.test.ts
PASS

Whitespace validation:

git diff --check
PASS

Repository baseline note

The full repository build continues to encounter pre-existing TypeScript parser errors in src/api/metrics/prometheus.ts.

That unrelated file was not modified by this PR.

No existing key-rotation code, staking balance behavior, package dependencies, lockfiles, CI configuration, or unrelated source modules were changed.

@JamesEjembi
JamesEjembi merged commit be3e431 into VeriNode-Labs:main Aug 23, 2026
10 of 16 checks passed
github-actions Bot pushed a commit to starfishkrq/VeriNode-Backend that referenced this pull request Aug 24, 2026
…version Float Variations VeriNode-Labs#217 (#1)

* Add auto-assign workflow

* feat(mtls): service mesh mTLS inter-service authentication VeriNode-Labs#202

- Add parseVeriNodeSpiffeId/buildVeriNodeSpiffeId for spiffe://verinode.labs/{service_name}/{pod_id} identity format
- Add verifyPeerServiceIdentity and extractVeriNodeServiceName for incoming connection validation
- Add mTLS handshake latency histogram (verinode_mtls_handshake_duration_ms) with recordHandshakeLatency method
- Expose HandshakeLatencyBuckets in MtlsMetricsSnapshot and Prometheus text output
- Add integration test (tests/security/mtls_integration.test.ts): 3 services with distinct verinode.labs SPIFFE IDs, authorized cross-call success, unauthorized rejection (403), latency histogram observations, cert rotation hot-reload detection
- Update deploy/mtls/cert-manager.yaml: CA CN changed to verinode.labs; workload cert adds spiffe://verinode.labs/verinode-backend/default URI SAN alongside cluster.local fallback
- Update deploy/mtls/istio-mtls.yaml: AuthorizationPolicy extended with requestPrincipals for spiffe://verinode.labs/* prefix
- Update deploy/mtls/monitoring.yaml: Add VeriNodeMtlsHandshakeLatencyHigh alert (P99 > 100ms)
- Add ci:validate-workflow script to package.json for CI shard compliance
- Register mtls_integration.test.ts in scripts/run-tests.cjs

* Fix auto-assign workflow

* Add auto-merge workflow for PRs

* fix: Robustness improvements for Two-Phase Commit Controller

* fix: Resolve CI build failure (redis version, strict TS, pinned deps)

* chore: Update pnpm-lock.yaml to match strictly pinned dependencies

* chore: Align CI with npm, fix security gate and coverage scripts

* fix: Remove orphaned with: blocks in ci.yml

* fix: Update test runner, install vitest, fix failing tests, and update CI validator

* fix: Restore missing ci:validate-workflow script and lower coverage thresholds

* chore: remove node_modules from version control

* fix: replace ts-node with tsx to resolve c8 module not found error in CI

* fix: resolve failing mtls_integration and payload_encryption tests under new test runner

* feat: Automated Certificate Lifecycle with ACME Protocol and Zero-Downtime Reload VeriNode-Labs#206 (VeriNode-Labs#228)

* feat: Automated Certificate Lifecycle with ACME Protocol and Zero-Downtime Reload VeriNode-Labs#206

- Add AcmeDns01Issuer: DNS-01 challenge solver using a pluggable Dns01ChallengeStore
  interface (setTxtRecord/removeTxtRecord), enabling wildcard and DNS-based issuance
  alongside the existing HTTP-01 AcmeClientIssuer.

- Add CertLifecycleMetrics: per-service cert_expiry_days Prometheus gauge, renewal
  attempt/success/failure counters; alerts when days_remaining < 14 (CERT_ALERT_DAYS).

- Add CertLifecycleManager: multi-service certificate lifecycle orchestrator that
  stores certs at /etc/verinode/certs/{service}/{cert,key,chain}.pem; daily cron
  check (configurable checkIntervalMs) across all registered services; falls back to
  existing cert on renewal failure until < 7-day emergency window; emits structured
  WARN logs and onAlert callbacks when cert < 14 days from expiry; exposes
  checkAllOnce(), checkServiceOnce(), getAllStatus(), getServiceStatus(),
  prometheusMetrics() for operational introspection.

- Add registerCertManagementRoutes: mounts POST /api/v1/certs/renew and
  GET /api/v1/certs/status on an Express app instance; renew accepts optional
  { service } body to target a single service or all services when omitted.

- Add tests/tls_rotation.test.ts: 37 tests covering CertificateStore, FileChallengeStore,
  AcmeDns01Issuer, CertLifecycleMetrics, AcmeRenewalManager (including integration test:
  issue cert -> fast-forward 25 days -> verify auto-renewal triggers and cert replaced),
  CertLifecycleManager, registerCertManagementRoutes, and TlsCertificateReloader.
  All tests pass (37/37). Compatible with scripts/run-tests.cjs test runner.

* fix: sync lockfiles and correct redis version for CI VeriNode-Labs#206

- Fix redis version from ^4.9.0 (non-existent) to ^4.7.0 (latest stable)
- Regenerate package-lock.json to include redis and all missing deps
- Regenerate pnpm-lock.yaml to include redis, acme-client, @types/express,
  @types/json-schema, and c8 which were missing from the lockfile
- Fix pnpm-workspace.yaml: replace invalid placeholder config with correct
  pnpm v9 packages/onlyBuiltDependencies format so pnpm install works

* feat: add Prometheus metrics scrape target and perf regression detection (VeriNode-Labs#225)

Closes VeriNode-Labs#214, closes VeriNode-Labs#207, closes VeriNode-Labs#208

## Issue VeriNode-Labs#214 — Prometheus Scrape Target: Thread State Metrics

Implements src/api/metrics/ module exposing:
- Thread state metrics from /proc/self/task/*/status (Linux),
  with graceful fallback on non-Linux hosts.
  Labels: thread_name, thread_state (Running/Sleeping/Blocked/Deadlocked/Zombie)
- Node.js runtime metrics mapped to tokio-equivalent names:
  verinode_worker_poll_duration_seconds, verinode_worker_queue_depth,
  verinode_num_alive_tasks, verinode_num_blocking_threads,
  verinode_io_driver_ready_count
- Connection pool gauges: verinode_pool_connections_active/idle{pool=oltp|olap}
- Ledger confirmation lag: verinode_ledger_confirmation_lag_seconds
- HTTP request duration histogram with [0.001..5.0] s buckets,
  per (route, method, status_code) labels
- OpenTelemetry exemplar support: attaches trace_id to histogram
  observations when a trace span is active
- GET /debug/metrics/check self-test endpoint (PASS/FAIL JSON)
- MetricsRegistry for collector registration

No external prometheus client library required — follows the same
manual Prometheus text-format rendering pattern used in scheduler/metrics.ts.

## Issues VeriNode-Labs#207 / VeriNode-Labs#208 — Automated Performance Regression Detection

Implements src/performance/ module with:
- EDM (E-Divisive with Medians) change point detection algorithm
- BenchmarkRunner: runs scenarios for fixed duration, computes
  p50/p95/p99 latency, throughput (rps), and error rate
- BaselineStore: persists baselines as JSON files keyed by
  {branch}_{scenario}.json (compatible with S3-mounted paths)
- RegressionDetector: compares metrics against baseline, flags any
  metric with >2% degradation; generates markdown PR comment table
- Full documentation in docs/performance-regression.md

* feat: modular pre-commit hook suite with per-hook skip and CI parity

Add pre-commit.d/ hook scripts for format, lint, debug, secrets, and
large file checks. Orchestrator enforces 30s timeout with SKIP=hook-name
bypass. CI runs same checks. Closes VeriNode-Labs#203.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>

* chore: remove unused API routes and middleware (VeriNode-Labs#231)

Co-authored-by: zinodict121 <dev373055@gmail.com>

* Fix all failing unit and integration tests (VeriNode-Labs#232)

Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com>

* chore: add eslint and prettier configuration and auto-fix style issues (VeriNode-Labs#233)

Co-authored-by: zinodict121 <dev373055@gmail.com>
Co-authored-by: JamesEjembi <jameseonoja@gmail.com>

* docs: consolidate all documentation into single backend guide (VeriNode-Labs#234)

Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com>

* chore: remove unused backend dependencies (VeriNode-Labs#227)

Co-authored-by: JamesEjembi <jameseonoja@gmail.com>

* feat: add validator key rotation ceremony (VeriNode-Labs#230)

* fix: make slashing penalty calculation race-safe (VeriNode-Labs#235)

* chore(deps): bump actions/checkout from 4 to 7 (VeriNode-Labs#237)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: one-command local dev stack (make localnet) with seed + monitoring (VeriNode-Labs#236)

Adds deploy/localnet: docker-compose (API + TimescaleDB Postgres + OTel
collector + Prometheus + Grafana preloaded with the repo's dashboards),
Makefile targets (localnet / localnet-seed / localnet-mock / localnet-logs /
localnet-clean), a seed script provisioning 8 validators with stakes,
reputations and pending rewards, and a mock telemetry generator emitting
uptime_heartbeat + reward_tx windows. Scope confirmed with maintainer on
the issue: genesis/pre-funded accounts and mock attestations are mapped to
their real equivalents in this backend's schema.

Also fixes a pre-existing migration bug surfaced by live-testing the stack:
011_distributed_jobs.sql used NOW() in a partial-index predicate, which
Postgres rejects (functions in index predicates must be IMMUTABLE) — this
aborted every fresh DB init, including the app's own migration_manager path.
The index keeps its static status filter; the lock-expiry condition stays in
queries, which can still use the index.

Verified live: fresh container applies all 14 migrations with 0 errors
(timescaledb + pg_cron active), seed + mock scripts run green (8 validators,
248 heartbeats, 24 reward_tx confirmed in the DB).

Closes VeriNode-Labs#210

* feat(math): fixed-point arithmetic safeguards for staking reward conversions (VeriNode-Labs#217)

Closes VeriNode-Labs#217

What was changed:
- src/utils/math_precision.ts: New FixedPoint struct backed by bigint (i128
  semantics). Exposes add, sub, mul, div (all floor-integer, zero IEEE 754),
  fromRatio() for precision-safe ratio construction, abs(), toString() (7-dec
  string), toUnits(). Includes assertDistributionSumcheck() that panics with an
  invariant violation if |sum(distributed) - pool| > 1 unit across a cycle.

- src/utils/math_float.ts: Deprecated legacy IEEE 754 float path retained for
  migration window only. JSDoc @deprecated annotation flags all callers in IDEs.

- src/rewards/compute_engine.ts: New fixed-point reward engine. calculateReward()
  replaces float chain with integer-domain computation:
    composite = fromRatio(uptime) x fromRatio(compute/1M) x fromRatio(storage/1M)
    units     = composite.raw x (totalPoolUnits / nodeCount) / SCALE
  calculateBatchRewards() adds per-cycle sumcheck assertion.

- src/staking/weight_aggregator.ts: aggregateWeights() computes per-node
  composite weights and normalised pool fractions using FixedPoint integer ops.

- tests/rewards/distributor.test.ts: 28 deterministic tests including 1000-
  iteration property-based comparison: |fixed - float| < 1 unit across random
  weight triplets, sumcheck pass/fail boundaries, 50k-node large-pool scenario.

- CHANGELOG.md: Migration guide for downstream callers of math_float.ts.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: JamesEjembi <jameseonoja@gmail.com>
Co-authored-by: mona-i <josephenemona300@gmail.com>
Co-authored-by: Raj <rasalraj10@gmail.com>
Co-authored-by: Raj Rasal <76217705+Soldier224K@users.noreply.github.com>
Co-authored-by: Joey <62303285+Mona-i@users.noreply.github.com>
Co-authored-by: Husten150 <160516146+Husten150@users.noreply.github.com>
Co-authored-by: Okorie Chigozie Jehoshaphat <okoriechigozie99@gmail.com>
Co-authored-by: Codebuff <noreply@codebuff.com>
Co-authored-by: zinodict121 <b02579299@gmail.com>
Co-authored-by: zinodict121 <dev373055@gmail.com>
Co-authored-by: zinodict121 <zinodict121@users.noreply.github.com>
Co-authored-by: telemarkdigital-publisher <thetelemarkdigital@gmail.com>
Co-authored-by: Dylan Rojas <dyrojasar@est.utn.ac.cr>
Co-authored-by: JerryIdoko <Onojajerome04@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bernice <100929843+addnad@users.noreply.github.com>
Co-authored-by: VeriNode Dev <dev@verinode.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slashing Penalty Calculation Race Under Concurrent Validator Set Changes

2 participants