🛡️ Aegis

Distributed Real-Time Client-Side Observability Platform

Aegis is a high-performance, self-hosted error observability platform engineered to ingest, deduplicate, and visualize high-volume client-side crashes with guaranteed durability. Built with a decoupled, stream-driven architecture, Aegis guarantees zero data loss by decoupling ingestion from storage through durable Redis Streams, consumer groups with atomic acknowledgment, and PostgreSQL Row-Level Security (RLS) multi-tenant isolation.

---

🏛️ Architecture

flowchart TD
    subgraph Client ["Client Layer"]
        SDK["@aegis/sentinel SDK"]
    end

    subgraph GatewayLayer ["Gateway & Ingestion Layer (:3001)"]
        GW["Express Gateway"]
        Auth["DSN Validator (SHA-256 Hash + Cache)"]
        RL["Atomic Rate Limiter (Token Bucket)"]
    end

    subgraph Broker ["Message Broker"]
        Stream[("Redis Streams\naegis:stream:events")]
        CG["Consumer Group\naegis_workers"]
        DLQ[("Dead Letter Queue\naegis_dlq")]
    end

    subgraph WorkerLayer ["Processing Worker"]
        Worker["Background Worker Consumer"]
        Dedup["Error Normalizer & SHA-256 Hasher"]
        Claim["XAUTOCLAIM Crash Recovery"]
    end

    subgraph Storage ["PostgreSQL 16 (The Vault)"]
        Partitions[("Partitioned Tables\nevents_pYYYY_MM_DD")]
        Issues[("Deduplicated Issues Table")]
        RLS["Fail-Closed Row-Level Security\n(SET LOCAL ROLE aegis_app)"]
    end

    subgraph UI ["Analytics Dashboard (:5173)"]
        SPA["React SPA (Vite + Recharts)"]
    end

    SDK -->|HTTP POST /api/ingest/capture| GW
    GW --> Auth
    Auth --> RL
    RL -->|XADD MAXLEN ~1M| Stream
    Stream --> CG
    CG -->|XREADGROUP| Worker
    Worker --> Dedup
    Dedup -->|Batch Upsert + INSERT| Storage
    Storage -->|COMMIT Success| Worker
    Worker -->|XACK Stream Message| Stream
    Worker -.->|Poison Pill Fallback| DLQ
    Claim -.->|Reclaim Unacked Messages| Stream
    SPA -->|Authenticated REST API| GW

Loading

Core Components

Component	Technology	Role
Sentinel SDK	Vanilla JS / ESM / UMD	Lightweight (< 5KB) client-side capture agent with beforeSend redaction and recursion guards.
Gateway	Node.js + Express	High-concurrency ingestion engine with Zod schema validation, SHA-256 DSN auth, and pipelined streaming.
Worker	Node.js	Asynchronous consumer group processor with SHA-256 issue deduplication, XACK on commit, and XAUTOCLAIM crash recovery.
Dashboard	React 18 + Vite	Real-time monitoring UI with project switching, KPI trends, error boundaries, and hourly incident volume charts.
Message Broker	Redis 7 Streams	Durable event log buffer with consumer groups, automatic trimming, and Dead Letter Queue (aegis_dlq).
The Vault	PostgreSQL 16	Partitioned analytical storage (events_pYYYY_MM_DD) with fail-closed Row-Level Security (RLS).

---

🚀 Quick Start

1. Environment Setup

Clone the repository and prepare your environment:

cp .env.example .env

2. Launch Infrastructure

Start PostgreSQL and Redis using Docker:

docker compose up -d postgres redis

3. Run Migrations & Start Services

# Run database migrations
npm run migrate

# Start all services concurrently (Gateway, Worker, Dashboard)
npm run dev

4. Access Aegis

Open your browser at http://localhost:5173.

• Default Admin Email: admin@aegis.dev
• Default Admin Password: admin123

---

📦 SDK Integration

Inject the Sentinel SDK into your web application to capture uncaught errors, unhandled promise rejections, and user breadcrumbs:

<script src="dist/aegis-sentinel.min.js"></script>
<script>
  Aegis.init({
    dsn: 'aegis_key_YOUR_PROJECT_API_KEY',
    environment: 'production',
    gatewayUrl: 'http://localhost:3001',
    beforeSend: function(payload) {
      // Optional: Redact sensitive PII before transmission
      if (payload.user && payload.user.email) {
        payload.user.email = payload.user.email.replace(/(?<=.).(?=.*@)/g, '*');
      }
      return payload;
    }
  });

  // Manual exception capture:
  try {
    performRiskyOperation();
  } catch (error) {
    Aegis.captureException(error);
  }
</script>

---

📊 Measured Performance Benchmarks

The benchmark harness (benchmark/run-benchmarks.js) executes real end-to-end traffic (1,000 HTTP requests, concurrency 25) against live Gateway, Redis Streams, Worker, and PostgreSQL instances.

Metric	Target	Measured Result	Status
Ingestion Success Rate	100.0%	100.0% (0% drop rate)	✅
Gateway Ingestion Throughput	> 1,000 req/s	1,256 req/s (single-node local)	✅
Gateway Response Latency (p50)	< 30 ms	15.7 ms	✅
Gateway Response Latency (p95)	< 50 ms	26.9 ms	✅
End-to-End Visibility Latency	< 2.0s (p99)	55.0 ms mean / 92.0 ms p95	✅
Test Suite Coverage	100% Workspaces Passing	36/36 tests passing (5 workspaces)	✅

Reproducing Benchmarks

# 1. Start infrastructure and services
docker compose up -d postgres redis
npm run migrate
node gateway/src/index.js &
node worker/src/index.js &

# 2. Run the automated benchmark harness
node benchmark/run-benchmarks.js

---

🛡️ Security & Reliability Properties

• At-Least-Once Delivery & Zero Data Loss: Events are written to Redis Streams. The Worker only calls XACK after the PostgreSQL transaction commits successfully. If a worker node crashes mid-batch, XAUTOCLAIM reassigns pending messages after 60 seconds.
• Poison-Pill Isolation (DLQ): Malformed payloads or unparseable entries are routed to aegis_dlq with failure metadata and acked from the primary stream to prevent pipeline starvation.
• Fail-Closed Multi-Tenant Row-Level Security: PostgreSQL Row-Level Security enforces strict tenant isolation on projects, issues, and events. Unauthenticated or missing tenant queries return 0 rows.
• Hashed Secret Storage: Project API keys are hashed via SHA-256 (api_key_hash). The raw API key is returned exactly once during generation or rotation (POST /api/projects/:id/rotate-key).
• Automated Partition Lifecycle: Events are stored in daily partitions (events_pYYYY_MM_DD). Partitions for today and tomorrow are pre-created automatically, while data older than 14 days is pruned via partition dropping.

---

Author: Anas Babari