Skip to content

techStack.md

Latest commit

 

History

History
353 lines (233 loc) · 20.9 KB

techStack.md

File metadata and controls

353 lines (233 loc) · 20.9 KB
Raw

Tech stack — engineering decisions, in detail

This case study is docOnly (no onChain contracts). The interesting engineering lives in two services: listenerEngine (multiChain event detection) and wallet (custodial key management). The entries below focus on those two plus a few crossCutting architectural choices.

Format per entry: What, Why, When, How, Alternatives, Why not those, Pros, Cons.

1 · ReorgSafe multiChain listenerEngine

What · A NestJS service that polls 5 chains (Ethereum, BSC, Polygon, Solana, Tron) for ERC20 / SPL / TRC20 Transfer events targeting escrow deposit addresses. PerChain confirmation buffer; sequential polling; PostgreSQL checkpoints; replay mode; RPC failover.

Why · RealTime deposit detection on multiple chains is the foundational requirement for any crypto escrow. Doing it correctly means handling four hard problems at once: (1) chain reorgs, (2) RPC failures and nodeState inconsistency, (3) catchUp after a service restart, and (4) avoiding doubleProcessing the same event across overlapping polls.

When · Continuously, while the service is running. Each chain has its own listener instance (evm.listener.ts, solana.listener.ts, tron.listener.ts) spawned by listener.service.ts.

How · PerChain polling loop:

  1. Read lastProcessedBlock from Postgres for this chain.
  2. Compute safeBlock = currentBlock - confirmationBuffer (e.g., currentBlock - 12 for ETH).
  3. If safeBlock <= lastProcessedBlock, sleep and retry.
  4. Otherwise, fetch logs from lastProcessedBlock + 1 to safeBlock, filtered to the contract addresses and the Transfer(address,address,uint256) topic.
  5. For each log, parse the from/to/amount, enrich with token metadata, publish to Kafka deposit.detected topic.
  6. After batch is published, update lastProcessedBlock = safeBlock in Postgres. Commit.
  7. Sleep for poll interval (1-5 seconds per chain), repeat.

The confirmation buffer is the reorg defense. By only processing blocks that are N confirmations deep, the listener never sees blocks that might be reverted by a reorg.

Alternatives

  • WebSocket subscriptions to chain RPC.
  • BlockByBlock processing without a buffer (process pending blocks).
  • ThirdParty indexer (The Graph, Alchemy Notify, QuickNode Streams).
  • OffChain data provider (Moralis, Covalent).

Why not those

  • WebSocket — works on EVM with caveats (subscriptions drop, reconnections lose events). Doesn't exist on Solana the same way. Forces the listener into reactive mode, which is harder to checkpoint and replay.
  • No buffer — invites processing pending blocks that get reorged out. Customer's deposit looks successful, then disappears.
  • ThirdParty indexer — adds a thirdParty trust dependency to the most critical path. Vendor outage = no deposits processed. Vendor data correctness = a black box.
  • OffChain provider — same trust dependency.

Pros · Pure pull model (idempotent — rePolling the same blocks is safe) · checkpointing makes restart trivial · vendorIndependent · perChain tuning (BSC reorgs deeper than ETH, so BSC needs a bigger buffer).

Cons · Latency: a deposit isn't detected until confirmationBuffer blocks pass. ETH at 12 blocks × 12s/block = ~2.5 minutes. Acceptable for escrow (no one is making a flashLoanStyle trade on it); not acceptable for highFrequency trading.

2 · AWS KMS direct encryption with no caching

What · Custodial private keys are encrypted with AWS KMS using Encrypt/Decrypt directly (not envelope encryption). Plaintext keys are NEVER cached in application memory beyond the lifetime of a single transfer operation. Each transfer triggers a fresh Decrypt call.

Why · Cached plaintext keys are vulnerable to memoryDisclosure attacks (heap dumps, swap files, JITSpray bugs). With no cache, the worstCase compromise (attacker reads process memory) reveals only keys currently being used — not the entire wallet table.

KMS direct encryption (vs envelope encryption with a data key) is the right choice when the payload is small (private keys are 32-64 bytes). Envelope encryption optimizes for large payloads where the perCall KMS cost would dominate.

When · Every wallet transfer (evmExecutor, solanaExecutor, tronExecutor). Each call: load encrypted key from Postgres, kms.decrypt(...), use the plaintext inMemory for the duration of the signing operation, then dereference (let GC reclaim).

How ·

// pseudoCode, paraphrased
class KmsService {
  async decryptKey(encryptedKey: Buffer, encryptionContext: Record<string, string>): Promise<Buffer> {
    return await this.withRetry(() =>
      this.kms.decrypt({
        CiphertextBlob: encryptedKey,
        EncryptionContext: encryptionContext, // bound to (service, user, depositId) tuple
      })
    );
  }
}

class PlatformKeyService {
  // NO CACHE. Every call hits KMS.
  async getKey(walletId: string, ctx: OperationContext): Promise<Buffer> {
    const row = await this.db.platformWallets.findUnique({ where: { id: walletId } });
    return this.kms.decryptKey(row.encryptedKey, {
      service: 'wallet',
      walletId,
      operationId: ctx.operationId,
      userId: ctx.userId,
    });
  }
}

Encryption context (an auditTrail binding) is included in every Encrypt/Decrypt call. KMS logs the context in CloudTrail — so every key decryption is auditable as "service X decrypted key Y for operation Z on behalf of user W". A retroactive audit can confirm every decrypt was legitimate.

Alternatives

  • Cached decrypted keys (inMemory map with TTL).
  • Envelope encryption (KMS encrypts a data key; data key encrypts the wallet key; cache the data key).
  • HSMStored keys (CloudHSM, never leave the HSM).
  • SelfManaged encryption (libsodium + KMSStored master key).

Why not those

  • Cached — defeats the entire security model. A memory disclosure leaks the cache.
  • Envelope — adds complexity for no real benefit at this payload size (32-64 bytes). KMS's perCall latency is ~10-30ms; that's amortized into the transfer latency anyway.
  • CloudHSM — significantly more expensive ($1000+/month minimum) and operationally heavier. Worth it for higherVolume products; overEngineered here.
  • SelfManaged — reImplements what KMS provides, with more attack surface.

Pros · No longLived plaintext keys in memory · CloudTrail audit per key use · KMS handles rotation and access control · standard pattern for AWSNative fintech.

Cons · PerTransfer latency includes a KMS roundTrip (~10-30ms). Acceptable for escrow; would be problematic for highFrequency operations. Also: AWS dependency (if KMS is unavailable, no transfers happen — same risk as any AWSNative service).

3 · Hot / funding / cold wallet tiering

What · Three classes of platformControlled wallets, with different security postures: hot (for routine outbound transfers), funding (for replenishing hot wallets from cold), cold (the bulk of the platform's reserves, signed by multisig with offline keys).

Why · HotWallet compromise is the single mostCited fintech failure. A wallet that signs many small transactions per hour is necessarily reachable from the operational systems — so its keys are necessarily exposed to any compromise of those systems. Limiting hotWallet balance bounds the loss. Cold storage holds the rest. Funding wallets are the intermediate — used to top up hot wallets in batched transfers, themselves protected by stricter access controls.

When · Configured at deployment time per chain. Routine escrow payouts go from the hot wallet for that chain. Once the hot wallet falls below a threshold, an ops alert fires; a manual (or partiallyAutomated) batched transfer from funding tops it up. ColdToFunding moves are multisigSigned, rare, and audited.

How · Three address sets in the platformWallets table per chain: role: 'hot' | 'funding' | 'cold'. The wallet service routes outbound transfers based on the destination amount + current hot balance — small transfers always hot; if the hot balance can't cover, transfer fails and triggers an ops alert.

Alternatives

  • One platform wallet per chain — simpler, allEggsOneBasket.
  • Pure hotOnly — no cold storage, all funds reachable from the application.
  • PerCustomer cold storage — each customer's balance has its own cold key.

Why not those

  • One wallet — a compromise of the hot key drains everything. Industry standard says don't.
  • Pure hot — same problem.
  • PerCustomer cold — operationally untenable at scale (thousands of cold keys to manage).

Pros · Bounded blastRadius per compromise · standard pattern (every reputable custodian uses it) · clear ops escalation when fundingToHot transfers are needed.

Cons · Two human ops events per "top up cold to funding" cycle (multisig signing). The dwell time for funds in the funding wallet between topUps is a smaller, timeBounded version of the hotWallet risk.

4 · PerChain executors with shared interface

What · Three transfer executors (evmExecutor, solanaExecutor, tronExecutor) implementing a shared IExecutor interface. Each handles the chainSpecific quirks of building, signing, broadcasting, and confirming a transfer.

Why · Each chain has its own SDK and gotchas. EVM uses ethers.js with gasPrice estimation. Solana uses @solana/web3.js and requires Associated Token Account (ATA) creation if the recipient hasn't received the SPL token before. Tron uses TronWeb with its own gasEquivalent (energy/bandwidth) cost model and SUN units (1 TRX = 1,000,000 SUN). Forcing them into one common executor would create a multiThousandLine god class with chainSpecific branching.

When · WalletService.transfer(...) dispatches to the appropriate executor based on the destination chain. The interface is uniform: transfer(from, to, amount, token, context).

How ·

  • EvmExecutor — uses ethers v6. Estimates gas, applies a 1.2x multiplier (configurable), broadcasts, polls getTransactionReceipt until status === 1 (success) or status === 0 (revert). Reverts surface as TransferFailed.
  • SolanaExecutor — checks if the recipient's Associated Token Account exists; if not, includes a createAssociatedTokenAccount instruction in the transaction. Builds + signs the transaction, sends with sendAndConfirmTransaction, parses the confirmation result.
  • TronExecutor — uses TronWeb. For TRC20: builds a trigger smart contract call. For TRX native: builds a regular transfer. Converts amount to SUN. Polls getTransactionInfo for confirmation.

Alternatives

  • Single universal executor with chain branches.
  • PluginBased executor with hotReload.
  • OffChain transaction service (e.g., Fireblocks, MPC providers).

Why not those

  • Universal — god class, hard to test perChain, hard to add a new chain.
  • PluginBased — overEngineered for 5 chains that change rarely.
  • OffChain custodian — adds a thirdParty dependency to the most critical path (custody). Defeats the "we hold the keys" property of being a custodian.

Pros · Each executor is testable in isolation · adding a new chain (e.g., Aptos, Sui) is "implement the IExecutor interface" · perChain quirks contained in their respective files.

Cons · Three implementations to maintain. A common feature (e.g., adding a perTransfer fee deduction) requires three code changes. Mitigated by a shared BaseExecutor for the parts that ARE common (retry, logging, KMS interaction).

5 · Postgres multiSchema (vs databasePerService)

What · Twelve NestJS services share one Postgres cluster. Each service has its own schema (escrow, wallet, etc.) with its own tables, indexes, and Prisma client. CrossService joins are forbidden.

Why · EachServiceItsOwnDatabase (DBPS) is the strict microservices ideal. In practice, it's operationally heavy: 12 Postgres instances to monitor, back up, version, and pay for. MultiSchema with strict ownership preserves the logical isolation (a service can only read/write its own schema's tables) while collapsing the ops footprint.

When · Each service's Prisma schema lives in services/<service>/prisma/schema.prisma and generates a client in services/<service>/generated/prisma/. The Postgres role for each service has GRANT on only its schema.

How · Postgres roles, schemas, and searchPath:

CREATE ROLE escrowService WITH LOGIN PASSWORD '...';
CREATE SCHEMA escrow AUTHORIZATION escrowService;
GRANT USAGE ON SCHEMA escrow TO escrowService;
-- No crossSchema grants.

Each service connects with its own role, sees only its own schema. Prisma migrations run perService against its own schema.

Alternatives

  • One DB per service — full DBPS isolation.
  • Single schema, shared tables — monolith DB pattern.
  • Polyglot persistence — different storage per service (MongoDB, Postgres, DynamoDB).

Why not those

  • One DB per service — operationally heavy.
  • Single schema — defeats service ownership boundaries.
  • Polyglot — adds complexity proportional to number of storage choices; each new tech adds operational burden.

Pros · One Postgres to operate · perService ownership preserved via roles · cheaper than DBPS · familiar SQL stack for all services.

Cons · CrossSchema joins are technically possible from a superuser — a misconfigured service could leak data from another. Mitigated by leastPrivilege roles and CI checks on Prisma schemas. Also: a runaway query in one service can affect clusterWide performance.

6 · Kafka outbox pattern via Prisma

What · State changes that need to publish a Kafka event do so atomically with the DB write. The pattern: a transaction writes to the domain table (e.g., escrows) AND inserts a row in outbox table; a separate publisher process reads outbox rows in order and publishes to Kafka, then marks them as published.

Why · "Update DB then publish Kafka" has two failure modes: DB succeeds, Kafka fails (event lost) or Kafka succeeds, DB fails (phantom event). The outbox pattern eliminates both by making the DB write + outbox insert one transaction.

When · Every Kafka producer in the service mesh. The escrow service uses prismaOutbox.adapter.ts and produceEvents.ts.

How ·

// PseudoCode (paraphrased from the actual NestJS service)
await this.prisma.$transaction(async (tx) => {
  await tx.escrow.update({ where: { id }, data: { state: 'funded' } });
  await tx.outbox.create({
    data: {
      topic: 'escrow.transitioned',
      payload: { escrowId: id, fromState: 'accepted', toState: 'funded' },
      idempotencyKey: `escrow-${id}-funded-${Date.now()}`,
    },
  });
});

// Separate publisher loop (Bull worker or NestJS scheduled task):
const pending = await this.prisma.outbox.findMany({
  where: { publishedAt: null }, orderBy: { createdAt: 'asc' }, take: 100,
});
for (const row of pending) {
  await this.kafka.produce(row.topic, row.payload);
  await this.prisma.outbox.update({ where: { id: row.id }, data: { publishedAt: new Date() } });
}

The outbox table guarantees atLeastOnce delivery. Idempotency on the consumer side (deduplicating by idempotencyKey) makes the system exactlyOnce from the consumer's POV.

Alternatives

  • Publish then commit — naive, has both failure modes mentioned above.
  • Transactional outbox via Debezium CDC — Debezium reads Postgres WAL and publishes to Kafka.
  • Sagas / event sourcing — events are the source of truth, derived state.

Why not those

  • Publish then commit — wrong.
  • Debezium CDC — adds Debezium as infrastructure. Powerful but heavy; the inProcess publisher is sufficient at our scale.
  • Event sourcing — much bigger architectural change. Worth considering for v2; overEngineering for v1.

Pros · Atomic DB+event semantics · simple to implement with Prisma's transaction support · idempotency built in via the idempotencyKey · publisher can be restarted without losing events.

Cons · Latency: events take 50-200ms to publish (one polling interval of the publisher). Acceptable for escrow; not acceptable for realTime market data. Also: the outbox table grows; a cleanup job archives published rows after N days.

7 · NestJS 10 with Prisma per service

What · Every service is a NestJS 10 app with its own Prisma schema and generated client. Standard NestJS patterns throughout: dependency injection, decorators for HTTP routes, Guards for auth, Interceptors for logging.

Why · NestJS gives the conventional structure a 12-service codebase needs (ExpressStyle with firstClass testing, dependency injection, and configuration). Prisma gives typeSafe Postgres access with migrations and a generated client per service.

When · Every service. New service creation involves: scaffold NestJS app, define Prisma schema, generate Prisma client, write the service's modules.

How · Standard NestJS module structure:

services/escrow/
├── prisma/
│   ├── schema.prisma
│   └── migrations/
├── src/
│   ├── modules/
│   │   ├── escrows/
│   │   │   ├── escrow.controller.ts
│   │   │   ├── escrow.service.ts
│   │   │   ├── escrowStateMachine.service.ts
│   │   │   └── escrow.module.ts
│   │   ├── milestones/
│   │   └── ...
│   ├── common/ (guards, interceptors, pipes)
│   ├── config/
│   └── main.ts
├── generated/
│   └── prisma/ (generated Prisma client)
└── package.json

Alternatives

  • Express + TypeORM — less opinionated.
  • Hono / Elysia — lighter, newer.
  • Spring Boot / Java — different language entirely.
  • Go (with gin or chi) — different language.

Why not those

  • Express — works but you reImplement most of what NestJS provides (DI, lifecycle, decorators).
  • Lighter frameworks — younger ecosystem, fewer integrations.
  • Java / Go — language change requires reSkilling.

Pros · Conventional codebase structure speeds onboarding · Prisma's typeSafe queries catch bugs at compile time · NestJS's DI testability is excellent.

Cons · NestJS has its own learning curve (decorators, modules, providers). For a small project the overhead isn't justified — for a 12-service mesh, it is.

8 · RealTime push via Socket.IO with Redis adapter

What · CustomerFacing realTime updates (escrow state changes, deposit confirmations, dispute notifications) push via Socket.IO. The Redis adapter (@socket.io/redis-adapter) lets multiple BFF instances share rooms and broadcast across the cluster.

Why · Customers expect "your deposit was confirmed" to show up in the UI without a page refresh. Polling would work but is wasteful (most pollers see no change). WebSocket + Socket.IO gives a clean eventDriven push.

When · After every state change that's customerRelevant. Service emits a Kafka event → notification service consumes → fanOuts to the right Socket.IO rooms (keyed by userId).

How · Notification service subscribes to relevant Kafka topics and rePublishes to Socket.IO. The Redis adapter coordinates broadcasts across all BFF replicas — a state change for userId: 123 reaches userId: 123's socket regardless of which BFF instance they're connected to.

Alternatives

  • ServerSent Events (SSE).
  • Polling at a reasonable interval (5-10s).
  • Push notifications only (mobileStyle, via FCM/APNS).

Why not those

  • SSE — oneWay (serverToClient), which is what we need, but Socket.IO has better browser support for fallback and reconnection.
  • Polling — wasteful; bad UX (a few seconds of staleness).
  • Push only — works on mobile but desktop browsers don't have firstClass push support.

Pros · Socket.IO handles reconnection, retry, exponential backoff out of the box · Redis adapter scales horizontally · falls back to longPolling if WebSocket is blocked.

Cons · WebSocket connections are stateful — load balancers need sticky sessions. Mitigated by the Redis adapter (rooms are coordinated crossInstance) but the initial connection still needs sticky.

9 · PerService Prisma + Kafka publishing observability

What · Each service exposes Prometheus metrics: request count, error rate, p50/p99 latency. Kafka consumers expose lag per topic. Aggregated logs to a central store, with request IDs flowing across services via OpenTelemetry tracing.

Why · 12 services × multiple instances each = a lot of moving parts. Without perService metrics, "the system is slow" is unactionable.

When · AlwaysOn. Metrics exposed at /metrics, scraped by Prometheus every 15s.

How · Standard NestJS + nestjsOtel + pino. Sentry for unhandled errors. Grafana dashboards per service.

Alternatives · DataDog, NewRelic, ELK stack.

Pros · OpenSource stack · perService ownership of dashboards.

Cons · SelfHosting the observability stack is its own ops burden. Many teams pay for managed services to avoid this.