Skip to content

README.md

Latest commit

 

History

History
232 lines (185 loc) · 11.6 KB

README.md

File metadata and controls

232 lines (185 loc) · 11.6 KB
Raw

Conduit — Event-Driven Fintech Backend Platform

Architecture: CQRS Messaging: Kafka License: MIT

Conduit is a high-performance, event-driven microservices ecosystem designed for real-time fintech data processing, anomaly detection, and automated remediation.

🚀 Quick Start

Prerequisites

  • Node.js >= 20.0.0
  • Docker & Docker Compose
  • Kafka Cluster (via Docker)

Installation & Development

# Install dependencies
npm install

# Start infrastructure (Kafka, Postgres, Redis, TimescaleDB)
docker-compose up -d

# Run all services in development mode
npm run dev:all

📖 Table of Contents

1. System Architecture

High-Level Diagram

┌──────────────────────────────────────────────────────────────────────────────────┐
│                                CLIENTS                                           │
│            (Web Dashboard · Mobile · Partner APIs · CLI)                         │
└─────────────┬──────────────────────────────────────────────┬────────────────────┘
              │ HTTPS (REST)                                 │ WSS (persistent)
              ▼                                              ▼
┌──────────────────────────────┐              ┌──────────────────────────────────┐
│   ① API GATEWAY  (:4000)    │              │  ⑦ WEBSOCKET SERVICE  (:4006)   │
│                              │              │                                  │
│  • Auth / Rate Limit        │              │  • Real-time Fan-out             │
│  • Request Routing          │              │  • Client Room Management        │
└──────┬────────────┬─────────┘              └──────────▲───────────────────────┘
       │            │                                   │ Consume
       │ HTTP       │ HTTP                              │
       ▼            ▼                                   │
┌────────────┐ ┌─────────────┐                          │
│ ② INGEST   │ │ ⑥ QUERY     │                          │
│   SERVICE  │ │   SERVICE   │                          │
│   (:4001)  │ │   (:4004)   │                          │
│            │ │             │                          │
│ • Write Gt │ │ • Read Model│                          │
└─────┬──────┘ └──────▲──────┘                          │
      │               │                                 │
      │ Produce       │ Read (Materialized)             │
      ▼               │                                 │
┌──────────────────────────────────────────────────────────────────────────────────┐
│                          ③ KAFKA EVENT BUS                                      │
│                                                                                  │
│  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│
│  │ conduit.events  │ │ conduit.metrics │ │ conduit         │ │ conduit         ││
│  │ .ingested       │ │ .computed       │ │ .incidents      │ │ .remediations   ││
│  └────────┬────────┘ └────────▲────────┘ └───────┬─────────┘ └───────▲─────────┘│
│           │                   │                  │                   │          │
└───────────┼───────────────────┼──────────────────┼───────────────────┼──────────┘
            │                   │                  │                   │
     ┌──────┴──────┐     ┌─────┴──────┐    ┌──────┴──────┐     ┌──────┴──────┐
     │ Consume     │     │ Produce    │    │ Consume     │     │ Produce     │
     ▼             ▼     │            │    ▼             ▼     │             │
┌──────────┐ ┌──────────┐│            │ ┌──────────┐ ┌──────────┐│             │
│ ④ METRIC │ │ ⑤ INCID. ││            │ │ ⑧ REMED. │ │ ⑤ INCID. ││             │
│  SERVICE │ │  SERVICE ││            │ │  SERVICE │ │  SERVICE ││             │
│  (:4003) │ │  (:4005) │┘            │ │  (Auto)  │ │ (CRUD)   │┘             │
└──────────┘ └──────────┘             │ └──────────┘ └──────────┘              │
                                      │        │                               │
                                      └────────┴───────────────────────────────┘

Strict Service Responsibilities

Service Primary Responsibility CQRS Role Communication
API Gateway Perimeter security, routing Entry Point HTTP (Inbound)
Ingestion Write validation, enrichment Command Kafka (Outbound)
Kafka Bus Persistent event log Message Hub Event Stream
Metrics Pure aggregation / windowing Processor Kafka (In/Out)
Incident Detection / Status CRUD Processor Kafka (In/Out) + HTTP (Reads)
Remediation Autonomous corrective actions Actor Kafka (In/Out)
Query High-perf materialized reads Query HTTP (Outbound)
WebSocket Live state synchronization Push WSS (Outbound)

2. Kafka Topic Design

Topic Registry

Topic Partitions Key Retention Compression Cleanup Policy
conduit.events.ingested 12 tenantId 7 days Snappy delete
conduit.metrics.computed 6 tenantId 3 days Snappy compact+delete
conduit.incidents.events 6 tenantId 30 days Snappy delete
conduit.ml.predictions 6 tenantId 14 days Snappy delete
conduit.remediations 3 tenantId 30 days Snappy delete

Consumer Groups

Group ID Service Subscribed Topics
conduit-metrics-group Metrics Service events.ingested
conduit-query-group Query Service events.ingested, metrics.computed, incidents.events, ml.predictions
conduit-incident-group Incident Service events.ingested, metrics.computed, ml.predictions
conduit-websocket-group WebSocket Service events.ingested, metrics.computed, incidents.events, remediations, ml.predictions
conduit-remediation-group Remediation Service incidents.events

3. Ingestion Service

The entry point for all raw events. Optimized for high throughput and durability.

Key Features

  • Persistent Producer: Zero TCP/TLS handshake overhead per request.
  • Redis Idempotency: SETNX with 24h TTL to prevent duplicate processing.
  • Batch Support: POST /ingest/batch for up to 500 events in one round-trip.
  • Snappy Compression: Optimized for network throughput.

4. Incident Service

Anomaly detection and lifecycle management engine.

State Machine

DETECTED → ACTIVE → RESOLVED

  • Detected: Initial trigger from threshold breach or ML anomaly.
  • Active: Acknowledged by operator or auto-escalated.
  • Resolved: Condition cleared or manual fix confirmed.

Detectors

  • Threshold Detector: Monitored via metrics.computed (Error Rate, P95, Success Rate).
  • ML Detector: Monitored via ml.predictions (Anomaly Score, Label Classification).

5. Remediation Service

Autonomous corrective actor that executes fixes when high-severity incidents are detected.

Execution Engine

  • Exponential Backoff: 2s × 2^n + jitter, max 3 retries.
  • Action Registry:
    • auto_rollback (Error Rate)
    • scale_out (Latency)
    • health_check_sweep (Success Rate)
    • adaptive_throttle (ML Anomaly)

6. Metrics Service

Pure aggregation pipeline for windowed statistical snapshots.

Aggregation Mechanics

  • Sliding Window: 1-minute window with p50/p95/p99 percentiles.
  • Dual-Write Path:
    • Hot Path: Instant Kafka snapshots for live dashboards.
    • Cold Path: TimescaleDB hypertable with tiered retention (7d raw → 90d agg).

7. Query Service

The read-side of the CQRS pattern. Optimized for sub-200ms materialized views.

Storage Split

  • PostgreSQL: Structured events, incidents, and remediations.
  • MongoDB: Schema-less ML predictions and feature vectors.
  • Redis: Hot-path metrics and 30s query cache.

8. WebSocket Service

Scalable real-time push service using Redis Pub/Sub for horizontal fan-out.

Features

  • Horizontal Scaling: Redis Pub/Sub fanz messages to all pods.
  • Backpressure: Drops clients exceeding 64KB buffer to protect the event loop.
  • Latency Tracking: End-to-end SLA tracking in every message envelope.

9. Tech Stack & Structure

Component Technology
Language Node.js (ESM)
Messaging Apache Kafka
Databases PostgreSQL, TimescaleDB, MongoDB, Redis
Testing Jest

Repository Structure

.
├── packages/             # Shared libraries (Kafka, Errors, Auth)
├── services/             # Microservices
│   ├── api-gateway/      # Entry Point
│   ├── ingestion-service/ # Command Side
│   ├── incident-service/  # Logic/State
│   ├── metrics-service/   # Processing
│   ├── remediation-service/ # Acting
│   ├── query-service/     # Query Side
│   └── websocket-service/ # Real-time Push
├── infra/                # Docker & DB Config
└── ...

Conduit — Hardened for Production.