chore: corruption-hardening ops (shutdown budget, journald logs, DuckDB spill)

Follow-up to the durable derivation queue. Three small prod-hardening changes against the 2026-06-06 incident's failure modes:

1. Shutdown budget made coherent: split into SHUTDOWN_WORKER_WAIT (30s, lets the worker finish its current derive) + SHUTDOWN_WRITE_WAIT (10s, in-flight raw upsert), summing to 40s < the new 45s stop_grace_period. Previously both phases used 25s each (worst case 50s > 30s grace -> Docker could SIGKILL mid-write).

2. API logging switched json-file -> journald so logs land in the host journal and SURVIVE container recreation. Root cause was unprovable on 2026-06-06 because the dead container's json-file logs vanished when it was recreated. docker logs + journalctl -t ohlc-datalake-api both still work.

3. DuckDB temp_directory set on the data volume so operations spill to disk past the 2GB memory_limit instead of growing until the cgroup OOM-kills the container mid-write (OOM-mid-checkpoint is the leading incident suspect on the 7.6GiB host).

Full suite 247 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: lucas-guerin-44

docker-compose.prod.yml

@@ -59,13 +59,18 @@ services:
           memory: 4g
           cpus: "2.0"
     restart: unless-stopped
-    # Let the FastAPI shutdown hook drain in-flight DuckDB writes before Docker SIGKILLs.
-    stop_grace_period: 30s
+    # Let the FastAPI shutdown hook drain the derivation worker + in-flight DuckDB
+    # writes before Docker SIGKILLs. Must exceed api.py's SHUTDOWN_WORKER_WAIT +
+    # SHUTDOWN_WRITE_WAIT budget (30 + 10 = 40s) — see that file.
+    stop_grace_period: 45s
+    # journald (not json-file) so the API's logs land in the host system journal
+    # and SURVIVE container recreation. After the 2026-06-06 incident the dead
+    # container's logs were gone (json-file dies with the container), making the
+    # crash unprovable. `docker logs` still works; so does `journalctl -t ohlc-datalake-api`.
     logging:
-      driver: json-file
+      driver: journald
       options:
-        max-size: "50m"
-        max-file: "5"
+        tag: "ohlc-datalake-api"
     networks:
       - datalake-network
 

src/api.py

@@ -17,7 +17,13 @@
 from src.config import validate_secrets
 from src.auth.auth import ScopedAuth
 
-SHUTDOWN_WRITE_WAIT_SECONDS = 25.0
+# Shutdown budget, split in two phases. The SUM must stay below the
+# docker-compose `stop_grace_period` (45s) or Docker SIGKILLs us mid-write — the
+# exact kill-during-checkpoint that risks DuckDB corruption. Phase 1 lets the
+# derivation worker finish its current task; phase 2 then waits out any in-flight
+# ingest's (short) raw upsert. 30 + 10 = 40s < 45s, leaving margin.
+SHUTDOWN_WORKER_WAIT_SECONDS = float(os.getenv("SHUTDOWN_WORKER_WAIT_SECONDS", "30"))
+SHUTDOWN_WRITE_WAIT_SECONDS = float(os.getenv("SHUTDOWN_WRITE_WAIT_SECONDS", "10"))
 from src.routes import (
     catalog_router,
     instruments_router,
@@ -96,8 +102,9 @@ def startup_event():
 @app.on_event("shutdown")
 def shutdown_event():
     """Stop the derivation worker, then block until in-flight writes finish so SIGTERM can't interrupt a transaction."""
-    # Stop the worker first so it can't start a NEW derive while we're draining.
-    derivation_queue.stop_worker(timeout=SHUTDOWN_WRITE_WAIT_SECONDS)
+    # Stop the worker first so it can't start a NEW derive while we're draining;
+    # this also waits out the worker's current derive task (phase 1 of the budget).
+    derivation_queue.stop_worker(timeout=SHUTDOWN_WORKER_WAIT_SECONDS)
     logger.info("Shutdown: waiting for in-flight writes", extra={"timeout_s": SHUTDOWN_WRITE_WAIT_SECONDS})
     acquired = _write_tx_lock.acquire(timeout=SHUTDOWN_WRITE_WAIT_SECONDS)
     if acquired:

src/core/datalake.py

@@ -34,9 +34,18 @@ def _get_shared_connection() -> duckdb.DuckDBPyConnection:
             if _db_connection is None:
                 _db_connection = duckdb.connect(str(DUCKDB_PATH))
                 _db_connection.execute(f"SET memory_limit = '{DUCKDB_MEMORY_LIMIT}'")
+                # Spill to disk on the data volume when an operation exceeds the
+                # memory_limit, instead of letting the process grow until the
+                # cgroup OOM-kills the container mid-write (a corruption risk, and
+                # the leading suspect for the 2026-06-06 incident on the RAM-tight
+                # host). The data volume has ample disk vs. the small memory cap.
+                tmp_dir = DUCKDB_PATH.parent / "duckdb_tmp"
+                tmp_dir.mkdir(parents=True, exist_ok=True)
+                _db_connection.execute(f"SET temp_directory = '{tmp_dir.as_posix()}'")
                 logger.info("DuckDB connection opened", extra={
                     "path": str(DUCKDB_PATH),
                     "memory_limit": DUCKDB_MEMORY_LIMIT,
+                    "temp_directory": str(tmp_dir),
                 })
     return _db_connection