chore: add devnet-triage skill and stop suffixed testnet dirs escaping gitignore - #428
Open
dimka90 wants to merge 2 commits into
Open
chore: add devnet-triage skill and stop suffixed testnet dirs escaping gitignore#428dimka90 wants to merge 2 commits into
dimka90 wants to merge 2 commits into
Conversation
Debugging a live devnet had become a round trip per query: report a symptom, get one command, paste a screenshot, get the next command. This collects the queries into one bundle per symptom, so a symptom costs one paste instead of a dozen. Bundles cover health, tick loop, finality and justification, aggregation, storage and pruning, memory, signature verification, and sync flapping, plus evidence capture before a stop -- container logs are the only copy in local mode and spin-node.sh --stop runs docker rm -f, which deletes them. The Traps section is the part worth keeping. It records the readings that have already cost time to rediscover: histograms whose buckets top out at 8 s and 4 s, so a p99 pinned there means "at least", not "that"; counters with only one label value, which make a dashboard error panel look clean when it is simply empty; cross-client metrics that share a name and not a meaning; max_over_time on a short window reading as a trend when the value oscillates; means dominated by no-op samples; and a single docker stats snapshot, which cannot represent a process that runs at 250% CPU during a proof and under 3% between them. Read-only by default. The contract is findings, evidence, proposed fix, then stop -- no restarts, no config edits, no aggregator toggling without approval for that specific action. Connection is through an SSH alias only. The host, user and key live in the operator's ~/.ssh/config and must never be written into this file: the alias name alone is meaningless without that config, which is what makes it safe to commit.
.gitignore covered testnet/ but not variants of it. A second local testnet generated as testnet-two/ was therefore untracked but not ignored, and it holds validator attestation and proposal secret keys plus node keys -- a git add -A would have committed them. Match testnet-*/ rather than the one name so the next variant is covered without another edit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two pieces of housekeeping. Neither touches client code, so neither needs a devnet run.
1.
devnet-triageskillDebugging a live devnet had become a round trip per query — report a symptom, get one command, paste a screenshot, get the next command. This collects the queries into one bundle per symptom, so a symptom costs one paste instead of a dozen.
Bundles: health, tick loop, finality and justification, aggregation, storage and pruning, memory, signature verification, sync flapping, plus evidence capture before a stop — container logs are the only copy in local mode and
spin-node.sh --stoprunsdocker rm -f, which deletes them.The Traps section is the part worth keeping. It records readings that have already cost time to rediscover:
max_over_timeon a short window reading as a trend when the value oscillateslean_proving_duration_secondsp50 reads ~80 ms while real proofs take secondsdocker statssnapshot, which cannot represent a process running at 250% CPU during a proof and under 3% between themRead-only by default. The contract is findings, evidence, proposed fix, then stop — no restarts, no config edits, no aggregator toggling without approval for that specific action.
On safety: connection is through an SSH alias only. Host, user and key live in the operator's
~/.ssh/configand the skill forbids writing them into any committed file. The alias name alone is meaningless without that config, which is what makes it safe to commit. Verified: no IP, hostname, username, or key path appears anywhere in the file.All 13 metric names the bundles reference were checked against a live devnet Prometheus and all resolve.
2.
.gitignore: ignore suffixed testnet directories.gitignorecoveredtestnet/but not variants. A second local testnet generated astestnet-two/was untracked but not ignored, and it holds validator attestation and proposal secret keys plus node keys — agit add -Awould have committed them.Now matches
testnet-*/so the next variant is covered without another edit. Verified withgit check-ignorethat the key files are now caught.