Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Once running:
| `GET /healthz/dependencies` | None | Shallow dependency probe — Postgres, Soroban RPC, Horizon, webhook queue (Redis). Cached for 5 s. Returns 200/207/503. |
| `GET /api/health/ready` | None | **Deep readiness check** — runs four parallel probes with 1-second timeouts each. Returns 200 when ready, 503 when unready. |
| `GET /api/indexer/health` | None | Indexer health — probes external dependencies (Postgres + Soroban RPC) and compares the persisted cursor against the chain tip. Returns `"ok"` / `"degraded"` / `"down"` with dependency statuses in `dependencies` and lag data in `data`. Always HTTP 200. Supports [ETag / conditional GET](#etag--conditional-get-caching). |
| `GET /api/recommendations/health` | None | Recommendations subsystem health — probes the two runtime dependencies the recommendations pipeline relies on (Postgres + Soroban RPC). Returns 200 when all pass, 503 when any is down. Response shape mirrors `GET /api/predictions/health`. |

### `GET /api/health/ready` response

Expand Down
152 changes: 152 additions & 0 deletions docs/recommendations-health.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
# `GET /api/recommendations/health`

Health probe for the `/api/recommendations` subsystem. Reports the status
of the two external dependencies the recommendations pipeline relies on.

---

## Why it exists

The recommendations endpoint surfaces personalised markets by querying
Postgres (for prediction history and market data) against a corpus of
markets that were indexed from the Soroban chain. If either dependency is
unavailable, personalised recommendations cannot be served. This probe lets
orchestrators and dashboards surface the root cause quickly.

---

## Request

```
GET /api/recommendations/health
```

No authentication required. No request body or query parameters.

Pass `X-Correlation-Id` to correlate the probe response with your
distributed-trace or alerting system:

```
X-Correlation-Id: my-trace-id-42
```

---

## Response

### 200 OK — all dependencies healthy

```json
{
"status": "ok",
"correlationId": "3a6d1f2c-...",
"checkedAt": "2026-07-28T19:27:42.000Z",
"dependencies": {
"database": { "status": "ok", "latencyMs": 4 },
"sorobanRpc": { "status": "ok", "latencyMs": 18 }
}
}
```

### 503 Service Unavailable — at least one dependency is down

```json
{
"status": "down",
"correlationId": "3a6d1f2c-...",
"checkedAt": "2026-07-28T19:27:42.000Z",
"dependencies": {
"database": { "status": "ok", "latencyMs": 3 },
"sorobanRpc": { "status": "down", "latencyMs": 5000, "error": "Soroban RPC unavailable" }
}
}
```

### Fields

| Field | Type | Description |
|---|---|---|
| `status` | `"ok"` \| `"down"` | Composite: `"ok"` only when **both** probes pass |
| `correlationId` | string | Echoes `X-Correlation-Id` header, or a generated UUID |
| `checkedAt` | ISO-8601 string | Timestamp of the probe run |
| `dependencies.database.status` | `"ok"` \| `"down"` | Postgres reachability |
| `dependencies.database.latencyMs` | number | Round-trip time in ms |
| `dependencies.database.error` | string? | Present only when `status = "down"` |
| `dependencies.sorobanRpc.status` | `"ok"` \| `"down"` | Soroban RPC reachability |
| `dependencies.sorobanRpc.latencyMs` | number | Round-trip time in ms |
| `dependencies.sorobanRpc.error` | string? | Present only when `status = "down"` |

---

## Probes

| Dependency | Probe method | Healthy signal |
|---|---|---|
| `database` | `SELECT 1` against the Postgres connection pool | Query resolves without error |
| `sorobanRpc` | `getLatestLedger()` against `SOROBAN_RPC_URL` | Response received without error |

Both probes run in parallel (`Promise.all`). The endpoint is **not** cached —
every request runs fresh probes. If you need caching, add a reverse-proxy or
sidecar cache in front of this path.

---

## HTTP status codes

| Code | Meaning |
|---|---|
| `200` | All dependency probes passed. |
| `503` | At least one dependency probe failed. The response body names which one. |
| `500` | An unexpected error was thrown inside a probe (not a graceful failure). Check logs with the `correlationId`. |

---

## Structured log events

Every probe run emits a `pino` log entry at level `info`:

```json
{
"level": 30,
"correlationId": "…",
"status": "ok",
"httpStatus": 200,
"elapsedMs": 22,
"database": "ok",
"sorobanRpc": "ok",
"msg": "recommendations_health_check_complete"
}
```

Unexpected errors emit at level `error`:

```json
{
"level": 50,
"correlationId": "…",
"err": { … },
"elapsedMs": 5,
"msg": "recommendations_health_probe_threw"
}
```

---

## Security

- No authentication required — the response contains no secrets or user data.
- In production, restrict access at the infrastructure level (internal ALB
rule, VPC-only routing, service-mesh policy, etc.) so external clients
cannot reach this path.

---

## Related endpoints

| Endpoint | Description |
|---|---|
| `GET /health` | Liveness check — no I/O |
| `GET /healthz/dependencies` | Shallow cached probe (all 4 deps, 5 s TTL) |
| `GET /api/health/ready` | Deep readiness for orchestrators |
| `GET /api/predictions/health` | Predictions-subsystem probe (same shape) |
| `GET /api/indexer/health` | Indexer health with cursor lag |
2 changes: 2 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import { redisConnection } from "./queue";
import { authRouter } from "./routes/auth";
import { recommendationsRouter } from "./routes/recommendations";
import { recommendationsHealthRouter } from "./routes/recommendations/health";
import { tagsRouter } from "./routes/tags";
import { auditRouter } from "./routes/audit";
import { marketsRouter } from "./routes/markets";
Expand Down Expand Up @@ -53,7 +54,7 @@
import { register } from "./metrics/registry";
import { connectWithRetry, closeDb, db } from "./db/client";
import { stopScheduler } from "./services/scheduler";
import { startIndexerHealthProbe, stopIndexerHealthProbe } from "./jobs/indexerHealthProbe";

Check failure on line 57 in src/index.ts

View workflow job for this annotation

GitHub Actions / lint-and-test

'stopIndexerHealthProbe' is defined but never used
import { indexerHealthRouter } from "./routes/indexer/health";
import { indexerCursorRouter } from "./routes/indexer/cursor";
import { WebhookWorker } from "./workers/webhookWorker";
Expand All @@ -67,7 +68,7 @@
import { reportsRouter } from "./routes/reports";
import { fingerprintRouter } from "./routes/fingerprint";
import { alertsRouter } from "./routes/alerts";
import { gracefulShutdown } from "./lifecycle/shutdown";

Check failure on line 71 in src/index.ts

View workflow job for this annotation

GitHub Actions / lint-and-test

'gracefulShutdown' is defined but never used


const docsEnabled = env.NODE_ENV !== "production" || process.env.ENABLE_DOCS === "true";
Expand Down Expand Up @@ -164,6 +165,7 @@
);

app.use("/api/auth", authRouter);
app.use("/api/recommendations/health", recommendationsHealthRouter);
app.use("/api/recommendations", recommendationsRouter);
app.use("/api/tags", tagsRouter);
app.use("/api/audit", auditRouter);
Expand Down
Loading
Loading