Skip to content

chore: add devnet-triage skill and stop suffixed testnet dirs escaping gitignore - #428

Open
dimka90 wants to merge 2 commits into
fix/aggregation-skip-visibilityfrom
chore/devnet-triage-skill
Open

chore: add devnet-triage skill and stop suffixed testnet dirs escaping gitignore#428
dimka90 wants to merge 2 commits into
fix/aggregation-skip-visibilityfrom
chore/devnet-triage-skill

Conversation

@dimka90

@dimka90 dimka90 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Two pieces of housekeeping. Neither touches client code, so neither needs a devnet run.

1. devnet-triage skill

Debugging a live devnet had become a round trip per query — report a symptom, get one command, paste a screenshot, get the next command. This collects the queries into one bundle per symptom, so a symptom costs one paste instead of a dozen.

Bundles: health, tick loop, finality and justification, aggregation, storage and pruning, memory, signature verification, sync flapping, plus evidence capture before a stop — container logs are the only copy in local mode and spin-node.sh --stop runs docker rm -f, which deletes them.

The Traps section is the part worth keeping. It records readings that have already cost time to rediscover:

  • histograms whose buckets top out at 8 s and 4 s, so a p99 pinned there means "at least", not "that"
  • counters with only one label value, which make a dashboard error panel look clean when it is merely empty
  • cross-client metrics sharing a name but not a meaning (ethlambda's finalisation counter counts attempts, gean's counts advances)
  • max_over_time on a short window reading as a trend when the value oscillates
  • means dominated by no-op samples — lean_proving_duration_seconds p50 reads ~80 ms while real proofs take seconds
  • a single docker stats snapshot, which cannot represent a process running at 250% CPU during a proof and under 3% between them

Read-only by default. The contract is findings, evidence, proposed fix, then stop — no restarts, no config edits, no aggregator toggling without approval for that specific action.

On safety: connection is through an SSH alias only. Host, user and key live in the operator's ~/.ssh/config and the skill forbids writing them into any committed file. The alias name alone is meaningless without that config, which is what makes it safe to commit. Verified: no IP, hostname, username, or key path appears anywhere in the file.

All 13 metric names the bundles reference were checked against a live devnet Prometheus and all resolve.

2. .gitignore: ignore suffixed testnet directories

.gitignore covered testnet/ but not variants. A second local testnet generated as testnet-two/ was untracked but not ignored, and it holds validator attestation and proposal secret keys plus node keys — a git add -A would have committed them.

Now matches testnet-*/ so the next variant is covered without another edit. Verified with git check-ignore that the key files are now caught.

Debugging a live devnet had become a round trip per query: report a symptom,
get one command, paste a screenshot, get the next command. This collects the
queries into one bundle per symptom, so a symptom costs one paste instead of
a dozen.

Bundles cover health, tick loop, finality and justification, aggregation,
storage and pruning, memory, signature verification, and sync flapping, plus
evidence capture before a stop -- container logs are the only copy in local
mode and spin-node.sh --stop runs docker rm -f, which deletes them.

The Traps section is the part worth keeping. It records the readings that
have already cost time to rediscover: histograms whose buckets top out at 8 s
and 4 s, so a p99 pinned there means "at least", not "that"; counters with
only one label value, which make a dashboard error panel look clean when it
is simply empty; cross-client metrics that share a name and not a meaning;
max_over_time on a short window reading as a trend when the value oscillates;
means dominated by no-op samples; and a single docker stats snapshot, which
cannot represent a process that runs at 250% CPU during a proof and under 3%
between them.

Read-only by default. The contract is findings, evidence, proposed fix, then
stop -- no restarts, no config edits, no aggregator toggling without approval
for that specific action.

Connection is through an SSH alias only. The host, user and key live in the
operator's ~/.ssh/config and must never be written into this file: the alias
name alone is meaningless without that config, which is what makes it safe to
commit.
.gitignore covered testnet/ but not variants of it. A second local testnet
generated as testnet-two/ was therefore untracked but not ignored, and it
holds validator attestation and proposal secret keys plus node keys -- a
git add -A would have committed them.

Match testnet-*/ rather than the one name so the next variant is covered
without another edit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant