Context
Raised in the review of #177 (see the inline comment on ai/advisor/models.py): the new news_article table in the AI Postgres grows without bound — RSS ingest is append-only forever, and the raw column stores the full original message next to the sanitized copy, roughly doubling storage per article.
The AI database lives on a fixed-size PVC. If it fills up, it does not just stop news ingest — it breaks recommendation persistence in the same database, i.e. the core AI feature. A full disk must never be able to take the system down.
Task
Make news storage fail-safe:
- Retention/pruning — enforce a hard cap with oldest-first eviction (e.g. delete-beyond-N or older-than-X on insert, or a periodic pruning job). A hard cap is the simplest design that guarantees bounded disk usage.
- Trim
raw — drop or truncate the duplicated raw payload once ingest is proven; it exists only for debugging.
- Graceful degradation — when writes fail because the DB is full, the consumer must dead-letter/skip instead of requeue-looping, so the RabbitMQ queue does not back up as a secondary failure.
- Consider an alert on AI-DB disk usage in the observability stack as a safety net.
Acceptance criteria
news_article disk usage is bounded regardless of uptime and feed volume.
- A simulated full-disk situation degrades news ingest only; recommendations keep working.
- Retention limits are configurable via Helm values / env vars.
Related: the untyped Vector column (no ANN index possible without a table rewrite) interacts with the retention decision — see the same review thread.
Context
Raised in the review of #177 (see the inline comment on
ai/advisor/models.py): the newnews_articletable in the AI Postgres grows without bound — RSS ingest is append-only forever, and therawcolumn stores the full original message next to the sanitized copy, roughly doubling storage per article.The AI database lives on a fixed-size PVC. If it fills up, it does not just stop news ingest — it breaks recommendation persistence in the same database, i.e. the core AI feature. A full disk must never be able to take the system down.
Task
Make news storage fail-safe:
raw— drop or truncate the duplicated raw payload once ingest is proven; it exists only for debugging.Acceptance criteria
news_articledisk usage is bounded regardless of uptime and feed volume.Related: the untyped
Vectorcolumn (no ANN index possible without a table rewrite) interacts with the retention decision — see the same review thread.