Proxmox host `server-1` memory overcommit OOM-killed the k8s-node VM — 2h55m baobab outage (2026-07-10 07:06 KST)

[haminprk]

Summary

On 2026-07-10 07:06 KST the Proxmox host server-1 ran out of physical RAM and the Linux kernel OOM-killer terminated the KVM guest process for the baobab k8s-node worker VM (the largest-RSS process on the box). The single k8s worker node backing our baobab cluster disappeared, and every orakl pod scheduled there went Pending/Terminating for ~2h55m. This is an infrastructure-layer (Proxmox host memory overcommit) problem, not an orakl application bug: the sum of VM memory reservations on server-1 (~72 GiB) exceeds the host's physical RAM (67.4 GiB), so the host had no headroom and killed a whole guest. It is distinct from the long-standing pod-level orakl-node memory leak, which self-heals via container restart and did not fire here.

Impact

• Outage window: ~2h55m. k8s-node went NotReady ("Kubelet stopped posting node status"); last heartbeat 07:03:08 KST, VM killed 07:06:41 KST.
• On-chain feeds stale: baobab Feeds BTC-USDT, IDRX-USDT, KAIA-USDT stopped updating on-chain for ~174 minutes. (Baobab testnet only — cypress is unaffected because it runs on a separate host and, for the node service, GKE.)
• Pods down: all orakl pods on k8s-node went Pending/Terminating for the full window (baobab schedules essentially the entire orakl workload onto this one worker).
• Post-reboot crash cascade Off-chain node #1 — Vault sidecar not injected: after the node rebooted, 9 pods entered CrashLoopBackOff with /vault/secrets/config: No such file. Root cause: vault-agent-injector was itself down when those pods were recreated, so its mutating webhook never injected the vault-agent sidecar. Vault itself was healthy (sealed: false). Fixed by a rollout restart of the affected deployments once the injector was back.
• Post-reboot crash cascade Adapter #2 — Loki compactor: loki-0 crashlooped with init compactor: failed to init delete store: timeout. The unclean shutdown corrupted its delete_requests boltdb. Renaming the directory let Loki recreate it.

Root cause

Host memory overcommit. server-1 has 67.4 GiB physical RAM but the VMs reserve ~72 GiB before the host OS is even counted:

VM	RAM
k8s-node	32 GiB
orakl-baobab-postgresql	13 GiB
k8s-ctr	9 GiB
baobab-vault	9 GiB
k8s-nfs	9 GiB
VM total	~72 GiB
host OS overhead	~4–6 GiB
committed	~76–78 GiB vs 67.4 GiB physical

That is ~4.6 GiB over-committed by the VMs alone, and ~9–11 GiB over once the host itself is included. When the host exhausted RAM the kernel OOM-killer selected the largest-RSS process, which is the 32 GiB k8s-node guest:

Jul 10 07:06:41 server-1 kernel: kvm invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Jul 10 07:06:42 server-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=qemu.slice,mems_allowed=0-1,global_oom,task_memcg=/qemu.slice/561.scope,task=kvm,pid=355514,uid=0
Jul 10 07:06:42 server-1 kernel: Out of memory: Killed process 355514 (kvm) total-vm:36243452kB, anon-rss:32211644kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:68480kB oom_score_adj:0

anon-rss: 32,211,644 kB ≈ 30.7 GiB, matching the 32 GiB k8s-node VM. constraint=CONSTRAINT_NONE + global_oom confirms this was host-wide RAM exhaustion, not a cgroup limit. The OOM score is roughly proportional to RSS, so a VM sized at 32 GiB is structurally the most likely victim on an overcommitted host — regardless of whether that VM is the actual cause of the pressure.

Contributing factors

1. Is the k8s-node VM's 32 GiB justified by the in-guest workload? Evidence is mixed — flagging the tension explicitly.

From the baobab cluster audit (orakl-baobab-admin@bisonai, node k8s-node, capacity 32,860,460Ki = 31.34 GiB, allocatable 32,758,060Ki = 31.24 GiB, 45 pods / 41 Running):

• Sum of pod memory requests = 4.96 GiB — only 15.9% of allocatable. Requests are the only thing the scheduler reserves; by that measure the VM is enormously oversized.
• Sum of pod memory limits = 17.97 GiB — 57.5% of allocatable. Limits (not requests) drive worst-case RSS, and even the theoretical everything-balloons-at-once ceiling is ~13 GiB below what the guest kernel sees (31.24 GiB) and ~14 GiB below the 32 GiB VM allocation.
• But 31 of 45 pods have no memory limit on any container (34 of 45 have ≥1 uncapped container). Those pods are unbounded — real usage is not captured by the 17.97 GiB limit sum.
• We could not measure actual in-guest usage: metrics-server is NOT installed on baobab. Both kubectl top pods -A and kubectl top nodes return error: Metrics API not available. All baobab figures above are declared requests/limits, not live RSS.

Tension worth calling out: the declared specs (17.97 GiB total limits, 4.96 GiB requests) do not justify a 32 GiB VM — yet the host OOM log shows the guest had actually touched ~30.7 GiB anon-rss. Those are not contradictory once you account for (a) guest kernel page cache, which the guest fills with free RAM and which appears to the host as guest anon-rss, (b) the 31 uncapped pods, and (c) KVM guest RSS ratcheting up to the high-water mark and rarely being reclaimed without ballooning. Without in-guest metrics we cannot attribute that 30.7 GiB between committed pod RSS, kernel page cache, and non-reclaimed guest memory. So "the VM is oversized" is supported by the declared workload but not yet proven against real steady-state usage — see Open Questions.

2. Monitoring stack is a large and mostly-uncapped footprint on the same worker.

• The single biggest concentrated balloon risk on baobab is monitoring/loki-chunks-cache-0 (memcached): limit 8 GiB but request only 0.5 GiB — a 16x request-to-limit gap and the only container on the baobab node with a ≥4 GiB limit. Everything else is capped ≤2.06 GiB or uncapped.
• loki-0, prometheus, grafana, promtail all have empty resources: {} on both clusters — no request and no limit — so the largest potential RAM consumers are uncapped and undercount the "total limits" exposure.
• Monitoring accounts for ~51% of all capped memory limits on the worker on both clusters. On cypress (where metrics ARE available) the same loki chunks-cache is live at 8,253 MiB (~84% of its 9.6 GiB cap) — confirming this component genuinely consumes multiple GiB, not just reserves them.

3. The known orakl-node leak did NOT cause this, and a pod limit only partially caps guest pressure.

• No OOMKilled containers anywhere on baobab. The pod-level orakl-node leak is real but self-heals: on cypress, orakl-node was cgroup-OOMKilled once on 2026-06-11 (restartCount=1) and orakl-vrf once on 2026-05-19, each recovering on restart. The most recent is ~29 days before this incident. This incident was a host-level global OOM, a different failure mode.
• baobab orakl-node deployment: single container, limit 2 GiB / request 1 GiB (Burstable), no ts-sidecar. cypress: limit 4 GiB / request 1 GiB, plus a ts-sidecar with resources: {} (unbounded).
• Does the pod memory limit cap guest RSS? Only partially. The limit does cgroup-OOM the leaking container and self-heal it, but it does not cap node/VM pressure because (a) unbounded sidecars/pods have no ceiling, and (b) on Burstable clusters requests ≪ limits, so several pods bursting toward their limits can exceed node RAM and drive node-level pressure a single pod's limit cannot prevent.

Secondary failures exposed by the outage

These are latent problems the reboot merely revealed; both deserve separate hardening tickets:

1. Vault-agent-injector webhook ordering. When vault-agent-injector is down, pods that (re)schedule are admitted without the vault-agent sidecar and silently start missing /vault/secrets/*, then CrashLoop. Nothing blocks or retries the injection. 9 pods hit this. Hardening options: make the injector's MutatingWebhookConfiguration failurePolicy: Fail for the affected namespaces (so pods are rejected rather than admitted secret-less), and/or gate orakl deployments on injector readiness. Trade-off to weigh: failurePolicy: Fail can itself block scheduling during an injector outage — needs discussion.
2. Loki compactor boltdb corruption on unclean shutdown. loki-0 could not recover its delete_requests store (init compactor: failed to init delete store: timeout) after the hard power loss. Recovery required manual directory rename. Consider moving the compactor/delete store to more durable storage or adding a startup repair/reset path.

Proposed actions

(a) Immediate — relieve server-1 overcommit

• Migrate one VM off server-1. Strongest candidate: orakl-baobab-postgresql (13 GiB); alternative baobab-vault (9 GiB). Live migration is preferred if the IP can remain static. — owner: @martin / infra team — effort: M (½–1 day incl. validation)
• Confirm post-migration headroom: after moving a VM, host committed RAM should sit comfortably below 67.4 GiB with room for host OS + page cache. — owner: @martin — effort: S

(b) Short-term — right-size and cap

• Right-size the k8s-node VM once true steady-state RSS is known (see Open Questions). Declared limits sum to 17.97 GiB; a 20–24 GiB VM may be sufficient and would remove the host-level overcommit even without a migration. — owner: @tom.cheese — effort: M
• Set memory limits on the 31 uncapped baobab pods (34 uncapped containers), prioritizing loki, prometheus, grafana, promtail, and the orakl TS microservices. Right-size loki-chunks-cache (8 GiB limit vs 0.5 GiB request). — owner: @tom.cheese — effort: M
• Install metrics-server on baobab so live usage is observable (it is already present on cypress). — owner: infra team — effort: S

(c) Preventive

• Host-level memory alerting on all Proxmox hosts: alert when committed VM reservations approach physical RAM and when host MemAvailable drops below a threshold — so overcommit is caught before the OOM-killer runs. — owner: @martin — effort: M
• Review host overcommit / KSM settings on server-1 (and disable or bound aggressive memory overcommit) so VM reservations cannot silently exceed physical RAM. — owner: infra team — effort: S–M
• Audit cypress for the same latent risk. cypress is not hard-overcommitted on paper (worker limits 21.17 GiB < 31.24 GiB allocatable; live usage 16.47 GiB), but 32 of 44 worker pods have no memory limit, and its host carries k8s-node (32G) + k8s-ctr (9G) + ... too. Verify the cypress host's physical RAM vs VM reservations. — owner: @martin — effort: S
• File the two secondary-failure hardening tickets (vault injector ordering, Loki boltdb). — owner: @tom.cheese — effort: S

Open questions

1. Migrate Postgres or Vault? orakl-baobab-postgresql (13 GiB) frees the most RAM; baobab-vault (9 GiB) is smaller but Vault's outage during this incident amplified the blast radius (sidecar injection failures). Which do we move, and does moving Vault off this host actually reduce coupling?
2. Can orakl-baobab-postgresql live-migrate while keeping a static IP? Infra flagged live migration as viable "if the IP can stay static" — needs confirmation for the DB specifically.
3. What is the true steady-state RSS of the k8s-node guest? The host saw ~30.7 GiB anon-rss but declared pod limits total only 17.97 GiB and requests 4.96 GiB. Until metrics-server is installed on baobab we cannot split that 30.7 GiB between committed pod memory, guest page cache, and non-reclaimed balloon — and therefore cannot finalize a right-sized VM value.
4. Do we want a broader memory-management pass? Infra's position is "we should do overall memory management" across both clusters — is that in scope for this ticket or a follow-up epic?

[haminprk]

Recurred within ~20 hours — still unmitigated

The same host memory-overcommit OOM fired again.

• First incident: 2026-07-10 07:06 KST (this issue).
• Recurrence: 2026-07-11 07:08 KST — the k8s-node kubelet last posted status at 2026-07-10 22:05 UTC, and the same idc-miko-kairos VM group (argocd, boot-api, delegator, loki-gateway-lb, orakl-dal, prometheus-grafana-lb) went offline together, the identical signature.
• Detected: 2026-07-13 09:45 KST. Baobab had been down ~50.7 hours (2.1 days) — on-chain Feeds (BTC/IDRX/KAIA) were stale for 3040+ minutes. It went unnoticed because baobab is testnet and the monitoring stack (grafana/loki) lives on the same downed host.

Recovery (2026-07-13, after infra rebooted the VM):

• Node came back Ready; Vault was already sealed: false (no unseal needed).
• Same post-reboot cascade as 7/10: 8 orakl deployments in CrashLoopBackOff with /vault/secrets/config: No such file (vault-agent sidecar not injected). Fixed with rollout restart.
• Same Loki init compactor: failed to init delete store: timeout boltdb corruption from the unclean shutdown; fixed by renaming the deletion dir again.
• On-chain feeds resumed (BTC 6s, IDRX/KAIA 51s).
• Note: the fsGroup recursive-chown fix (Aggregator related updates #593) is merged, so loki no longer stalls for 20+ min on chown — but the boltdb corruption on hard power-loss is unaffected and recurs every time.

The point

The 7/10 recovery was a band-aid; the root cause (VMs reserving ~72 GiB on a 67.4 GiB host) was never addressed, so it recurred in ~20 hours and then sat dead for 2 days. Every hard OOM-kill also risks corrupting Postgres / Vault / etcd on the same host, not just Loki.

The immediate action in this issue — migrate one VM off server-1 (postgresql 13G or vault 9G) — needs to actually happen. Until then this will keep recurring. Also worth adding: a basic liveness alert on baobab on-chain feed staleness so a 2-day outage can't go unnoticed again.