Keep 45 daily validation logs and delete the rest - #43
Merged
Conversation
Three remote branches are neither merged into main nor known to be obsolete, so git branch --merged will never retire them and the question comes back to whoever next tidies the repository. They belong to the af8fb02 line, whose four pull requests were absorbed by a separate reconciliation rather than merged directly. main is BELIEVED to contain their substance. Nobody has proven it, and this note says so plainly rather than implying the matter is settled. Deleting a remote branch that is not an ancestor of main is not reversible for anyone else, and keeping one costs nothing, so the trade favours leaving them. Proving equivalence is archaeology that needs judgement per branch, and nobody needs these branches today. The next person now meets a recorded fact instead of a puzzle, and knows what would have to be true before deleting is safe.
The daemon wrote one validation log per day and never removed any of them, so the directory grew for as long as the daemon ran. Measured on a live machine: 2.4 GB across 20 daily files going back to 2026-07-16, with a single noisy day at 587 MB. I searched the tree for retention, pruning, or a maximum age and there was none. Unbounded growth with nobody at the keyboard is a defect whatever today's number is. Disk is shared, so fabric exhausting it does not degrade fabric, it degrades the machine. FORTY-FIVE IS DERIVED, NOT ROUNDED. Retention has to outlast a month away from the machine, or a fault in the first week is deleted before the only person who investigates it gets home. That is a floor near 31 days. Two more weeks covers the gap between returning and looking. At the observed 8.8 to 10.3 MB per day that is roughly 420 MB. A test pins the number and asserts it exceeds a month-long trip, so moving it requires saying why. FAILING OPEN WOULD BE THE WORST OUTCOME, so an unparseable override falls back to the bound rather than to unbounded. On an unattended machine a typo would otherwise restore the defect silently and look like the fix never worked. An explicit 0 still disables deletion, for an operator who would rather spend disk than lose history. The resolved value is recorded in diagnostic_logging_init so the running configuration is checkable rather than assumed. THE FILENAME FORMAT IS UNCHANGED, and I verified that rather than trusting the builder's documentation: I ran a daemon on a scratch home and confirmed it still writes validation.log.<date>. A changed name would have broken every existing grep, including the ones this network already uses. WHAT THIS DOES NOT DO. It bounds the file COUNT, which is what stops indefinite growth. It does not bound bytes, because one day has reached 587 MB. Capping that is a question about log volume, not retention, and it is not solved here. Logs written before this bound are not reclaimed; that is an operator's call. 201 lib tests green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Approved by cos. Fabric never deleted a validation log.
Measured, not assumed
On a live machine:
tracing_appender::rolling::dailyrotates and deletes nothing, and there is no retention, pruning, or max-age logic anywhere in the tree.cos checked the urgency honestly and corrected me: that volume has 836 GiB free, so 2.4 GB is ~0.3% and nobody is close to being hurt. The work still stands — unbounded growth with nobody at the keyboard is a defect whatever today's number is, and disk is shared, so fabric exhausting it degrades the machine, not just fabric.
45 is derived, not rounded
Retention has to outlast a month away from the machine, or a fault in the first week is deleted before the only person who investigates it gets home. That is a floor near 31 days; two more weeks covers the gap between returning and looking. At the observed rate that is roughly 420 MB.
A test pins the number and asserts it exceeds a month-long trip, so moving it requires changing an assertion and saying why.
Failing open would be the worst outcome
An unparseable
FABRIC_LOG_RETENTION_DAYSfalls back to the bound, not to unbounded. On an unattended machine a typo would otherwise restore the defect silently and look like the fix never worked. An explicit0still disables deletion, for an operator who would rather spend disk than lose history.The resolved value is recorded in
diagnostic_logging_init, so the running config is checkable rather than assumed.Filename format verified, not trusted
Swapping
rolling::dailyfor theBuildercould have changed the filename. I ran a daemon on a scratch home and confirmed it still writesvalidation.log.<date>. A changed name would have broken every existing grep, including ones this network already uses.What this does not do
It bounds the file count, which is what stops indefinite growth. It does not bound bytes — one day reached 587 MB — and capping that is a question about log volume, not retention. Logs written before this bound are not reclaimed; that is an operator's call.
Documented plainly in the README and CHANGELOG, because silently removing a file someone might want is its own hazard.
Ordering note
This lands before enabling
pathwatchon purpose. Pathwatch roughly doubles the daily log rate, so enabling it first would have shipped a disk problem while fixing an observability one.Tests
201 lib tests green (4 new), plus lifecycle and provisioning. Changed files pass rustfmt.