Skip to content

feat(page-cache): reclaim stray /tmp mission trees to cut billed Railway footprint#2454

Merged
sukria merged 2 commits into
mainfrom
koan/minimize-railway-footprint-reclaim
Jul 19, 2026
Merged

feat(page-cache): reclaim stray /tmp mission trees to cut billed Railway footprint#2454
sukria merged 2 commits into
mainfrom
koan/minimize-railway-footprint-reclaim

Conversation

@sukria

@sukria sukria commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Since v0.8.0 the Railway memory graph plateaus near 1 GB while real process memory is tiny. Root cause is not a leak — it is cgroup memory.current page-cache accounting.

Diagnosed live on a fresh idle instance (2026-07-19):

Metric Value
anon (real private RAM) 78 MB
per-process RSS (ps) ~124 MB
billed memory.current ~600–900 MB

The reclaim log showed file dropping either to ~95 MB (fully reclaimed) or plateauing at ~663 MB — a single mission /tmp tree pinning ~570 MB of out-of root page cache that held for ~4.5h until the age-gated post-mission sweep deleted the files. That is the "stable at 1 GB for hours" an operator observes.

The universal reclaim (#2374) only fadvises pages under its roots (instance/ + venv + scratch + project workdirs); the stray /tmp mission trees (pytest-of-*, test-koan*, koan-*, jest_rs) were never covered.

Changes

  • default_reclaim_roots() also sweeps the stray /tmp mission trees matched
    by cleanup.extra_tmp_globsown-uid dirs only (read-only fadvise; skip
    other users' /tmp to avoid permission-denied churn). Their clean pages are now
    dropped immediately, decoupling billed footprint from the sweep's deletion latency.
  • page_cache_reclaim.idle_interval_s default 900 → 180s so boot and
    between-tick page cache never sits billed for long.
  • Spec contract (specs/components/agent-loop.md), instance.example/config.yaml,
    and docs/operations/memory-footprint.md updated; new test coverage for the
    glob / dir-only / own-uid filters.

The ultimate backstop for any out-of-root cache (and for kernel slab, which fadvise cannot touch) remains a cgroup memory limit on the service — noted in the docs.

Architectural declaration

This extends the durable default_reclaim_roots() contract in specs/components/agent-loop.md — changed contract-first, then the code made to conform.

Checklist

  • Architectural change — modifies a durable design contract
    (specs/components/agent-loop.md), declared above and reviewed contract-first.
  • Tests pass (test_page_cache.py, test_config.py, test_run_finally_reclaim.py)
  • make lint clean

🤖 Generated with Claude Code

…print

The universal page-cache reclaim (#2374) only drops clean pages under its
roots (instance/ + venv + scratch + project workdirs). Mission subprocesses
read large files into the page cache from stray /tmp trees (pytest-of-*,
test-koan*, koan-*, jest_rs) the roots never cover, so those pages stay billed
on the cgroup's memory.current until the age-gated post-mission sweep deletes
the files — hours later.

Observed live 2026-07-19 on a fresh idle instance: anon 78 MB / RSS ~124 MB
(healthy), but billed memory.current ~600-900 MB, with a mission tree pinning
~570 MB of out-of-root file cache for ~4.5h (the "stable at 1 GB for hours"
operators see). No leak — pure cgroup page-cache accounting.

- default_reclaim_roots() now also sweeps the same stray /tmp trees the tmp
  sweep targets (cleanup.extra_tmp_globs), own-uid dirs only (read-only fadvise;
  skip other users' trees to avoid permission-denied churn).
- Lower page_cache_reclaim.idle_interval_s default 900 -> 180 so boot and
  between-tick cache never sits billed for long.

Architectural: extends the default_reclaim_roots() contract in
specs/components/agent-loop.md (declared below).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot requested a review from Koan-Bot July 19, 2026 16:19
@railway-app
railway-app Bot temporarily deployed to koan-alexis / production July 19, 2026 17:01 Inactive
@sukria

sukria commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator Author

@sukria-koan0 fix failing CI tests

/usage_tracker.py                   146      5    97%   85-87, 282-283
app/utils.py                           765     72    91%   36, 355-356, 484-485, 500-501, 569-570, 587-589, 601-604, 608-609, 630-631, 633, 638-641, 650-653, 659-660, 719-720, 737-749, 776, 804-806, 876, 1028, 1031-1032, 1046, 1122-1123, 1128-1129, 1161-1162, 1164, 1168, 1188-1189, 1255, 1319, 1333-1334, 1341, 1416-1417, 1483, 1497, 1516-1517
app/version.py                          25      0   100%
app/workspace_discovery.py              47      0   100%
app/worktree_manager.py                209     15    93%   162-163, 274-276, 304, 311, 318, 332, 341, 372, 422, 447, 451-452
------------------------------------------------------------------
TOTAL                                52303   9396    82%
=========================== short test summary info ============================
FAILED tests/test_page_cache.py::test_stray_tmp_roots_ignores_non_tmp_patterns - AssertionError: assert [PosixPath('/...est-of-koan')] == []
  
  Left contains one more item: PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw3/test_stray_tmp_roots_ignores_n0/pytest-of-koan')
  
  Full diff:
  - []
  + [
  +     PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw3/test_stray_tmp_roots_ignores_n0/pytest-of-koan'),
  + ]
= 1 failed, 16995 passed, 1 skipped, 1 xfailed, 2 warnings in 132.01s (0:02:12) =
Error: Process completed with exit code 1.

CI's pytest basetemp lives under /tmp/pytest-of-runner, so the prior
test's tmp_path pattern started with /tmp/ and passed the guard,
returning the dir instead of []. Use a fixed non-/tmp pattern and spy
on glob to prove the guard short-circuits before globbing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@railway-app
railway-app Bot temporarily deployed to koan-alexis / production July 19, 2026 17:05 Inactive
@sukria

sukria commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator Author

@sukria-koan0 review (CI is now full green)

@sukria-koan0

Copy link
Copy Markdown
Collaborator

@sukria: @sukria-koan0 fix failing CI tests

Resolved. Failure came from pytest rooting its tmpdir under /tmp/pytest-of-*, so the guard's own glob matched. Fixed test now uses a /var/tmp/* pattern and spies on glob to prove the /tmp/ prefix guard short-circuits before globbing.

@sukria-koan0

Copy link
Copy Markdown
Collaborator

PR Review — feat(page-cache): reclaim stray /tmp mission trees to cut billed Railway footprint

Solid, well-scoped billing-hygiene fix. Merge-ready.

Strengths:

  • Root cause (out-of-root /tmp page-cache residuals) is diagnosed with live evidence and the fix is narrowly targeted — _stray_tmp_roots() reuses the existing cleanup.extra_tmp_globs list rather than inventing a parallel config surface.

  • Correct safety posture: strictly read-only fadvise, os.lstat + S_ISDIR skips symlinks so traversal never escapes /tmp, and the own-uid guard mirrors the sweep to avoid permission churn on shared hosts.

  • Priority ordering (stray-tmp before big project trees) matches the diagnosis that the stray tree is the dominant contributor.

  • Durable spec change under specs/components/agent-loop.md is declared architectural + contract-first per repo discipline; docs and instance.example/config.yaml updated in the same PR.

  • New tests cover the non-/tmp short-circuit, dir-only, and own-uid filters via mocked boundaries (behavior, not source inspection). Prior CI failure (pytest tmpdir matching /tmp/pytest-of-*) is resolved by using a /var/tmp/* pattern.

  • One suggestion only: the "in lockstep" comment overstates coupling — the reclaim omits the sweep's direct-/tmp-child guard (benign since reclaim is read-only).



Checklist

  • Read-only / safe filesystem traversal (symlink + uid guards)
  • Durable-spec change declared architectural
  • New behavior covered by behavior-focused tests
  • Docs + example config kept in sync
  • No privacy leaks on added lines
  • Config default change is backward-compatible (operator-overridable)

Automated review by Kōan (Claude · model opus) HEAD=3f80075 2 min 28s

@sukria-koan0 sukria-koan0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tip

No blocking issues found — ready to merge.

@sukria
sukria marked this pull request as ready for review July 19, 2026 17:14
@sukria
sukria merged commit 5c257a3 into main Jul 19, 2026
8 checks passed
@sukria
sukria deleted the koan/minimize-railway-footprint-reclaim branch July 19, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants