Conduit is a high-performance, event-driven microservices ecosystem designed for real-time fintech data processing, anomaly detection, and automated remediation.
- Node.js >= 20.0.0
- Docker & Docker Compose
- Kafka Cluster (via Docker)
# Install dependencies
npm install
# Start infrastructure (Kafka, Postgres, Redis, TimescaleDB)
docker-compose up -d
# Run all services in development mode
npm run dev:all- 1. System Architecture
- 2. Kafka Topic Design
- 3. Ingestion Service
- 4. Incident Service
- 5. Remediation Service
- 6. Metrics Service
- 7. Query Service
- 8. WebSocket Service
- 9. Tech Stack & Structure
┌──────────────────────────────────────────────────────────────────────────────────┐
│ CLIENTS │
│ (Web Dashboard · Mobile · Partner APIs · CLI) │
└─────────────┬──────────────────────────────────────────────┬────────────────────┘
│ HTTPS (REST) │ WSS (persistent)
▼ ▼
┌──────────────────────────────┐ ┌──────────────────────────────────┐
│ ① API GATEWAY (:4000) │ │ ⑦ WEBSOCKET SERVICE (:4006) │
│ │ │ │
│ • Auth / Rate Limit │ │ • Real-time Fan-out │
│ • Request Routing │ │ • Client Room Management │
└──────┬────────────┬─────────┘ └──────────▲───────────────────────┘
│ │ │ Consume
│ HTTP │ HTTP │
▼ ▼ │
┌────────────┐ ┌─────────────┐ │
│ ② INGEST │ │ ⑥ QUERY │ │
│ SERVICE │ │ SERVICE │ │
│ (:4001) │ │ (:4004) │ │
│ │ │ │ │
│ • Write Gt │ │ • Read Model│ │
└─────┬──────┘ └──────▲──────┘ │
│ │ │
│ Produce │ Read (Materialized) │
▼ │ │
┌──────────────────────────────────────────────────────────────────────────────────┐
│ ③ KAFKA EVENT BUS │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│
│ │ conduit.events │ │ conduit.metrics │ │ conduit │ │ conduit ││
│ │ .ingested │ │ .computed │ │ .incidents │ │ .remediations ││
│ └────────┬────────┘ └────────▲────────┘ └───────┬─────────┘ └───────▲─────────┘│
│ │ │ │ │ │
└───────────┼───────────────────┼──────────────────┼───────────────────┼──────────┘
│ │ │ │
┌──────┴──────┐ ┌─────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐
│ Consume │ │ Produce │ │ Consume │ │ Produce │
▼ ▼ │ │ ▼ ▼ │ │
┌──────────┐ ┌──────────┐│ │ ┌──────────┐ ┌──────────┐│ │
│ ④ METRIC │ │ ⑤ INCID. ││ │ │ ⑧ REMED. │ │ ⑤ INCID. ││ │
│ SERVICE │ │ SERVICE ││ │ │ SERVICE │ │ SERVICE ││ │
│ (:4003) │ │ (:4005) │┘ │ │ (Auto) │ │ (CRUD) │┘ │
└──────────┘ └──────────┘ │ └──────────┘ └──────────┘ │
│ │ │
└────────┴───────────────────────────────┘
| Service | Primary Responsibility | CQRS Role | Communication |
|---|---|---|---|
| API Gateway | Perimeter security, routing | Entry Point | HTTP (Inbound) |
| Ingestion | Write validation, enrichment | Command | Kafka (Outbound) |
| Kafka Bus | Persistent event log | Message Hub | Event Stream |
| Metrics | Pure aggregation / windowing | Processor | Kafka (In/Out) |
| Incident | Detection / Status CRUD | Processor | Kafka (In/Out) + HTTP (Reads) |
| Remediation | Autonomous corrective actions | Actor | Kafka (In/Out) |
| Query | High-perf materialized reads | Query | HTTP (Outbound) |
| WebSocket | Live state synchronization | Push | WSS (Outbound) |
| Topic | Partitions | Key | Retention | Compression | Cleanup Policy |
|---|---|---|---|---|---|
conduit.events.ingested |
12 | tenantId |
7 days | Snappy | delete |
conduit.metrics.computed |
6 | tenantId |
3 days | Snappy | compact+delete |
conduit.incidents.events |
6 | tenantId |
30 days | Snappy | delete |
conduit.ml.predictions |
6 | tenantId |
14 days | Snappy | delete |
conduit.remediations |
3 | tenantId |
30 days | Snappy | delete |
| Group ID | Service | Subscribed Topics |
|---|---|---|
conduit-metrics-group |
Metrics Service | events.ingested |
conduit-query-group |
Query Service | events.ingested, metrics.computed, incidents.events, ml.predictions |
conduit-incident-group |
Incident Service | events.ingested, metrics.computed, ml.predictions |
conduit-websocket-group |
WebSocket Service | events.ingested, metrics.computed, incidents.events, remediations, ml.predictions |
conduit-remediation-group |
Remediation Service | incidents.events |
The entry point for all raw events. Optimized for high throughput and durability.
- Persistent Producer: Zero TCP/TLS handshake overhead per request.
- Redis Idempotency:
SETNXwith 24h TTL to prevent duplicate processing. - Batch Support:
POST /ingest/batchfor up to 500 events in one round-trip. - Snappy Compression: Optimized for network throughput.
Anomaly detection and lifecycle management engine.
DETECTED → ACTIVE → RESOLVED
- Detected: Initial trigger from threshold breach or ML anomaly.
- Active: Acknowledged by operator or auto-escalated.
- Resolved: Condition cleared or manual fix confirmed.
- Threshold Detector: Monitored via
metrics.computed(Error Rate, P95, Success Rate). - ML Detector: Monitored via
ml.predictions(Anomaly Score, Label Classification).
Autonomous corrective actor that executes fixes when high-severity incidents are detected.
- Exponential Backoff: 2s × 2^n + jitter, max 3 retries.
- Action Registry:
auto_rollback(Error Rate)scale_out(Latency)health_check_sweep(Success Rate)adaptive_throttle(ML Anomaly)
Pure aggregation pipeline for windowed statistical snapshots.
- Sliding Window: 1-minute window with p50/p95/p99 percentiles.
- Dual-Write Path:
- Hot Path: Instant Kafka snapshots for live dashboards.
- Cold Path: TimescaleDB hypertable with tiered retention (7d raw → 90d agg).
The read-side of the CQRS pattern. Optimized for sub-200ms materialized views.
- PostgreSQL: Structured events, incidents, and remediations.
- MongoDB: Schema-less ML predictions and feature vectors.
- Redis: Hot-path metrics and 30s query cache.
Scalable real-time push service using Redis Pub/Sub for horizontal fan-out.
- Horizontal Scaling: Redis Pub/Sub fanz messages to all pods.
- Backpressure: Drops clients exceeding 64KB buffer to protect the event loop.
- Latency Tracking: End-to-end SLA tracking in every message envelope.
| Component | Technology |
|---|---|
| Language | Node.js (ESM) |
| Messaging | Apache Kafka |
| Databases | PostgreSQL, TimescaleDB, MongoDB, Redis |
| Testing | Jest |
.
├── packages/ # Shared libraries (Kafka, Errors, Auth)
├── services/ # Microservices
│ ├── api-gateway/ # Entry Point
│ ├── ingestion-service/ # Command Side
│ ├── incident-service/ # Logic/State
│ ├── metrics-service/ # Processing
│ ├── remediation-service/ # Acting
│ ├── query-service/ # Query Side
│ └── websocket-service/ # Real-time Push
├── infra/ # Docker & DB Config
└── ...
Conduit — Hardened for Production.