feat(cache): serve stale articles on upstream failure

Author: AlisinaDevelo

.env.example

@@ -8,6 +8,7 @@ PORT=3000
 # LOG_LEVEL=info
 # GNEWS_BASE_URL=https://gnews.io/api/v4
 # HTTP_TIMEOUT_MS=15000
+# STALE_CACHE_TTL_SEC=3600
 # SHUTDOWN_TIMEOUT_MS=10000
 # RATE_LIMIT_MAX=120
 # RATE_LIMIT_WINDOW_MS=60000

CHANGELOG.md

@@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Provider circuit breaker for repeated GNews failures, including Prometheus circuit event metrics and `503` short-circuit responses.
 - OpenAPI-generated TypeScript client types, a small v1 `NewsApiClient` wrapper, and CI drift checks for generated client output.
 - Root capability document that advertises v1 routes, legacy routes, docs, and observability endpoints.
+- Stale-on-error article cache fallback with `meta.cache=stale`, stale cache TTL configuration, and Prometheus stale cache metrics.
 
 ### Changed
 

README.md

@@ -9,16 +9,16 @@ A search runs through a small, explicit pipeline — each stage is a separate, t
 1. **Validate** — query params are checked before any network call: `query` is required, `count` is clamped (≤100), `lang`/`country` must be ISO two-letter codes, `from`/`to` must parse as ISO 8601, and `sortBy` ∈ {`publishedAt`, `relevance`}. Bad input fails fast with `400` instead of wasting an upstream call or quota.
 2. **Cache** — parameters are normalized into a deterministic key and read through a pluggable store (in-memory by default, Redis when `REDIS_URL` is set). Cache failures are logged/metriced but do not fail article requests; identical in-flight misses in the same process share one upstream call.
 3. **Upstream** — on a miss, GNews is called with a hard timeout; transport/provider failures surface as `502`, and repeated failures open a short circuit that returns `503` without amplifying the outage.
-4. **Observe** — each step emits structured Pino logs (carrying `x-request-id`), Prometheus counters (cache hit/miss, upstream outcome, latency histogram), and optional OpenTelemetry spans.
+4. **Observe** — each step emits structured Pino logs (carrying `x-request-id`), Prometheus counters (cache hit/miss/stale fallback, upstream outcome, latency histogram), and optional OpenTelemetry spans.
 5. **Respond** — legacy endpoints return raw article arrays, while `/api/v1/*` returns `{ data, meta }` envelopes with request/cache metadata and structured error bodies.
 
 Full diagram and component notes live in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 
 ## Production-oriented features
 
 - **Security:** [Helmet](https://helmetjs.github.io/) headers, configurable rate limiting, optional `TRUST_PROXY` for correct client IPs behind a load balancer, optional **`CLIENT_API_KEYS`** + `X-API-Key` on `/api/*`.
-- **Reliability:** Upstream HTTP timeouts, response validation, `502` for provider/transport failures, provider circuit breaker with `503` short-circuiting, graceful shutdown on `SIGTERM` / `SIGINT`.
-- **Observability:** JSON logs via [Pino](https://getpino.io/), `x-request-id`, **`GET /metrics`** ([Prometheus](https://prometheus.io/) text format), cache hit/miss/error/coalescing + upstream latency/circuit metrics, and optional **OpenTelemetry** traces to OTLP (`OTEL_EXPORTER_OTLP_*`).
+- **Reliability:** Upstream HTTP timeouts, response validation, stale-on-error cache fallback, `502` for provider/transport failures, provider circuit breaker with `503` short-circuiting, graceful shutdown on `SIGTERM` / `SIGINT`.
+- **Observability:** JSON logs via [Pino](https://getpino.io/), `x-request-id`, **`GET /metrics`** ([Prometheus](https://prometheus.io/) text format), cache hit/miss/stale/error/coalescing + upstream latency/circuit metrics, and optional **OpenTelemetry** traces to OTLP (`OTEL_EXPORTER_OTLP_*`).
 - **Kubernetes-style probes:** `GET /health` (liveness), `GET /ready` (readiness when the API key is configured).
 - **Supply chain:** `npm audit` in CI; **SPDX SBOM** artifacts; Docker builds with **SBOM + provenance**; **dependency review** on PRs; optional **SLSA-style lockfile attestation** on `main`; lockfile-only installs.
 - **Contract:** OpenAPI at **`GET /openapi.yaml`** (also on disk as [docs/openapi.yaml](docs/openapi.yaml)).
@@ -95,7 +95,7 @@ Base path: `/api`. Machine-readable schema: **`GET /openapi.yaml`** · source fi
 | `GET` | `/health` | Liveness: `{ "status": "ok", "uptime": number }`. |
 | `GET` | `/ready` | Readiness; `503` if `GNEWS_API_KEY` missing (non-test). |
 | `GET` | `/openapi.yaml` | OpenAPI 3 document (`application/yaml`). |
-| `GET` | `/metrics` | Prometheus metrics (skips rate limit), including HTTP totals, cache hit/miss/error/coalescing counts, and upstream latency. |
+| `GET` | `/metrics` | Prometheus metrics (skips rate limit), including HTTP totals, cache hit/miss/stale/error/coalescing counts, and upstream latency. |
 | `GET` | `/api/v1/articles` | Versioned search. Returns `{ data, meta }`, including normalized filters, cache status, and `requestId`. |
 | `GET` | `/api/v1/articles/search` | Alias for versioned search. |
 | `GET` | `/api/v1/articles/title/:title` | Exact title match with `{ data, meta }`, else structured `404`. |

docs/ARCHITECTURE.md

@@ -24,8 +24,8 @@ flowchart LR
 ### A search, end to end
 
 The cache is an optimization, not a dependency: hits skip the provider, misses are
-coalesced per process, and cache failures fall through to the upstream rather than failing
-the request.
+coalesced per process, cache failures fall through to the upstream rather than failing
+the request, and stale copies can serve reads when the provider fails.
 
 ```mermaid
 sequenceDiagram
@@ -44,8 +44,12 @@ sequenceDiagram
   else miss (read fail falls through here too)
     Svc->>Svc: coalesce identical in-flight misses
     Svc->>GNews: provider adapter: GET /api/v4/search (timeout)
-    GNews-->>Svc: normalized articles  (502 on provider/transport error)
+    GNews-->>Svc: normalized articles
     Svc->>Cache: set(key, articles, TTL 600s)
+    Svc->>Cache: set(stale key, articles, longer TTL)
+  else upstream failure and stale copy exists
+    Svc->>Cache: get(stale key)
+    Cache-->>Svc: stale articles
   end
   Svc-->>API: articles
   API-->>Client: 200 + results (legacy) or { data, meta } (v1)
@@ -86,10 +90,12 @@ Article arrays are stored per normalized search key with a **600-second** TTL (`
 
 The service treats the cache as a quota and latency optimization, not as a hard dependency. Read failures fall through to the upstream provider, write failures return the upstream response without caching it, and both paths emit warning logs plus cache error metrics. Within a single process, concurrent misses for the same normalized key are coalesced so only the first request calls the provider.
 
+Successful upstream searches write both a fresh cache key and a longer-lived stale key. The fresh key protects latency and quota under normal conditions; the stale key is only read after an upstream failure. Versioned responses expose this through `meta.cache=stale`.
+
 ## Provider Circuit Breaker
 
 The GNews provider adapter tracks consecutive provider failures. After `UPSTREAM_CIRCUIT_FAILURE_THRESHOLD` failures, it opens a cooldown window (`UPSTREAM_CIRCUIT_COOLDOWN_MS`) and returns `503` without making another upstream call. After the cooldown, one request is allowed through; success closes the circuit, while another failure opens it again.
 
 ## Metrics
 
-`src/metrics/register.ts` exports a single Prometheus registry used by `/metrics`. HTTP middleware records response counts, while `newsService` and the provider adapter record cache hits/misses/errors/coalesced misses, upstream request outcomes, upstream latency buckets, and circuit breaker events.
+`src/metrics/register.ts` exports a single Prometheus registry used by `/metrics`. HTTP middleware records response counts, while `newsService` and the provider adapter record cache hits/misses/errors/coalesced/stale misses, upstream request outcomes, upstream latency buckets, and circuit breaker events.

docs/OPERATIONS.md

@@ -10,6 +10,7 @@
 | `LOG_LEVEL` | `info` (non-test) | Pino level (`trace`–`fatal`, or `silent`). |
 | `GNEWS_BASE_URL` | `https://gnews.io/api/v4` | Upstream provider base URL. Override only for local integration tests, benchmarks, or compatible provider mocks. |
 | `HTTP_TIMEOUT_MS` | `15000` | Outbound GNews request timeout (max `60000`). |
+| `STALE_CACHE_TTL_SEC` | `3600` | Longer-lived stale article cache TTL used only as an upstream-failure fallback (min effective value `>600`, max `86400`). |
 | `UPSTREAM_CIRCUIT_FAILURE_THRESHOLD` | `3` | Consecutive provider failures before the circuit opens. |
 | `UPSTREAM_CIRCUIT_COOLDOWN_MS` | `30000` | How long to short-circuit provider calls after the circuit opens (max `300000`). |
 | `SHUTDOWN_TIMEOUT_MS` | `10000` | Force-exit if `server.close` does not finish. |
@@ -40,6 +41,7 @@
 - Cache keys include normalized search parameters: query, count, `lang`, `country`, `from`, `to`, and `sortBy`.
 - Cache reads/writes are non-fatal for article searches. If the cache backend is unavailable, the service logs a warning, increments cache error metrics, falls through to GNews on read failure, and still returns the upstream response on write failure.
 - Identical in-flight misses are coalesced per process, so concurrent requests for the same normalized search wait on one upstream provider request.
+- Successful searches are also written to a longer-lived stale cache key. If a later fresh miss hits an upstream failure and stale data is available, `/api/v1/*` returns `meta.cache=stale` with a `200` response instead of surfacing the provider outage.
 
 On shutdown the server closes the Redis connection when that backend was used.
 
@@ -50,13 +52,13 @@ On shutdown the server closes the Redis connection when that backend was used.
 | Metric | Labels | Purpose |
 |--------|--------|---------|
 | `http_requests_total` | `method`, `status_code` | HTTP response count. |
-| `news_cache_events_total` | `result=hit|miss|error|coalesced` | Cache lookup and in-flight coalescing behavior for article searches. |
-| `news_cache_errors_total` | `operation=get|set` | Cache backend errors that were tolerated by falling through to upstream or returning an uncached upstream response. |
+| `news_cache_events_total` | `result=hit|miss|error|coalesced|stale` | Cache lookup, stale fallback, and in-flight coalescing behavior for article searches. |
+| `news_cache_errors_total` | `operation=get|set|get_stale|set_stale` | Cache backend errors that were tolerated by falling through to upstream, returning an uncached upstream response, or skipping stale fallback. |
 | `news_upstream_requests_total` | `outcome=success|error|invalid_payload` | GNews provider request outcomes. |
 | `news_upstream_request_duration_seconds` | `outcome=success|error|invalid_payload` | GNews provider request latency histogram. |
 | `news_upstream_circuit_events_total` | `event=opened|short_circuit|half_open|closed` | Provider circuit breaker state transitions and short-circuited requests. |
 
-Use cache hit rate and coalesced miss counts to understand quota protection, cache error metrics to detect Redis/backend trouble, upstream latency/error metrics to separate provider trouble from local API trouble, and circuit events to see when repeated provider failures are being shed locally.
+Use cache hit rate and coalesced miss counts to understand quota protection, stale counts to see when provider trouble is being hidden by cached data, cache error metrics to detect Redis/backend trouble, upstream latency/error metrics to separate provider trouble from local API trouble, and circuit events to see when repeated provider failures are being shed locally.
 
 ## Docker
 

docs/openapi.yaml

@@ -705,7 +705,7 @@ components:
           additionalProperties: true
         cache:
           type: string
-          enum: [hit, miss, coalesced]
+          enum: [hit, miss, coalesced, stale]
         requestId:
           type: string
 

src/cache/store.ts

@@ -6,7 +6,7 @@ const TTL_SEC = 600;
 
 export type CacheStore = {
   get(key: string): Promise<unknown | undefined>;
-  set(key: string, value: unknown): Promise<void>;
+  set(key: string, value: unknown, ttlSec?: number): Promise<void>;
 };
 
 let singleton: CacheStore | null = null;
@@ -18,8 +18,8 @@ function createMemoryStore(): CacheStore {
     async get(key: string) {
       return c.get(key);
     },
-    async set(key: string, value: unknown) {
-      c.set(key, value);
+    async set(key: string, value: unknown, ttlSec = TTL_SEC) {
+      c.set(key, value, ttlSec);
     },
   };
 }
@@ -46,8 +46,8 @@ function createRedisStore(url: string): CacheStore {
         return undefined;
       }
     },
-    async set(key: string, value: unknown) {
-      await client.setex(key, TTL_SEC, JSON.stringify(value));
+    async set(key: string, value: unknown, ttlSec = TTL_SEC) {
+      await client.setex(key, ttlSec, JSON.stringify(value));
     },
   };
 }

src/client/openapi-types.ts

@@ -285,7 +285,7 @@ export interface components {
                 [key: string]: unknown;
             };
             /** @enum {string} */
-            cache: "hit" | "miss" | "coalesced";
+            cache: "hit" | "miss" | "coalesced" | "stale";
             requestId: string;
         };
         ArticleSearchEnvelope: {

src/services/newsService.ts

@@ -6,8 +6,9 @@ import { logger } from "../logger";
 import { newsProvider } from "../providers/gnewsProvider";
 
 const inFlightSearches = new Map<string, Promise<ArticleSearchResult>>();
+const STALE_CACHE_TTL_SEC = resolveStaleCacheTtlSec();
 
-export type ArticleSearchCacheStatus = "hit" | "miss" | "coalesced";
+export type ArticleSearchCacheStatus = "hit" | "miss" | "coalesced" | "stale";
 
 export interface ArticleSearchResult {
   articles: Article[];
@@ -26,6 +27,15 @@ function searchCacheKey(options: ArticleSearchOptions): string {
   });
 }
 
+function staleCacheKey(cacheKey: string): string {
+  return `${cacheKey}:stale`;
+}
+
+function resolveStaleCacheTtlSec(): number {
+  const raw = Number(process.env.STALE_CACHE_TTL_SEC ?? 3_600);
+  return Number.isFinite(raw) && raw > 600 ? Math.min(Math.floor(raw), 86_400) : 3_600;
+}
+
 async function readCachedArticles(
   store: CacheStore,
   cacheKey: string
@@ -47,6 +57,24 @@ async function readCachedArticles(
   return undefined;
 }
 
+async function readStaleCachedArticles(
+  store: CacheStore,
+  cacheKey: string
+): Promise<ArticleSearchResult | undefined> {
+  try {
+    const cached = await store.get(staleCacheKey(cacheKey));
+    if (cached !== undefined) {
+      cacheEventsTotal.inc({ result: "stale" });
+      return { articles: cached as Article[], cache: "stale" };
+    }
+  } catch (err) {
+    cacheErrorsTotal.inc({ operation: "get_stale" });
+    logger.warn({ err }, "stale cache get failed; returning upstream error");
+  }
+
+  return undefined;
+}
+
 async function writeCachedArticles(
   store: CacheStore,
   cacheKey: string,
@@ -58,14 +86,32 @@ async function writeCachedArticles(
     cacheErrorsTotal.inc({ operation: "set" });
     logger.warn({ err }, "cache set failed; returning upstream response without caching");
   }
+
+  try {
+    await store.set(staleCacheKey(cacheKey), articles, STALE_CACHE_TTL_SEC);
+  } catch (err) {
+    cacheErrorsTotal.inc({ operation: "set_stale" });
+    logger.warn({ err }, "stale cache set failed; returning upstream response without stale copy");
+  }
 }
 
 async function fetchArticlesFromUpstream(
   options: ArticleSearchOptions,
   store: CacheStore,
   cacheKey: string
 ): Promise<ArticleSearchResult> {
-  const articles = await newsProvider.search(options);
+  let articles: Article[];
+  try {
+    articles = await newsProvider.search(options);
+  } catch (err) {
+    const stale = await readStaleCachedArticles(store, cacheKey);
+    if (stale) {
+      logger.warn({ err }, "upstream failed; returning stale cached articles");
+      return stale;
+    }
+    throw err;
+  }
+
   await writeCachedArticles(store, cacheKey, articles);
   return { articles, cache: "miss" };
 }
@@ -84,7 +130,7 @@ export const searchArticles = async (
   if (existing) {
     cacheEventsTotal.inc({ result: "coalesced" });
     const result = await existing;
-    return { ...result, cache: "coalesced" };
+    return { ...result, cache: result.cache === "miss" ? "coalesced" : result.cache };
   }
 
   const search = fetchArticlesFromUpstream(options, store, cacheKey);

test/app.test.ts

@@ -252,6 +252,27 @@ describe("app", () => {
     expect(mockGet).toHaveBeenCalledTimes(1);
   });
 
+  it("returns stale cached articles when upstream fails after a fresh miss", async () => {
+    const staleArticles = sampleArticles.slice(0, 1);
+    setCacheStoreForTests({
+      async get(key) {
+        return key.endsWith(":stale") ? staleArticles : undefined;
+      },
+      async set() {
+        return undefined;
+      },
+    });
+    mockGet.mockRejectedValueOnce(new axios.AxiosError("timeout"));
+
+    const q = `stale-${Math.random().toString(36).slice(2)}`;
+    const res = await request(app).get(`/api/v1/articles?query=${encodeURIComponent(q)}&count=2`);
+
+    expect(res.status).toBe(200);
+    expect(res.body.data).toEqual(staleArticles);
+    expect(res.body.meta.cache).toBe("stale");
+    expect(mockGet).toHaveBeenCalledTimes(1);
+  });
+
   it("coalesces identical in-flight cache misses", async () => {
     mockGet.mockImplementationOnce(
       () =>