shadow-gossipsub: select muxer and discovery (static | kad-dht)#316
Merged
Conversation
The shadow experiment was hardcoded to yamux + static. Expose muxer, discovery and start_sleep on ExpConfig and wire them into shadow.yaml: kad-dht restores the bootstrap-0 anchor host + peer NODE_ROLE/SERVICE env (Shadow resolves it by hostname). Make multi-shadow-gossipsub sweep muxer and node count via SHADOW_MUXERS/SHADOW_SIZES/SHADOW_DISCOVERY so it can run the nim-libp2p regression matrix under Shadow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Long matrix names (nodes_1000__muxer_..._disc_kad-dht) pushed the derived shadow-<run_id>-reader pod name past the k8s 63-char limit, so log collection failed. Shorten get_name_from_params and cap run_id at 45 chars. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single bootstrap-0 anchor kept the default MAXCONNECTIONS=250, so at 1000 nodes only ~250 could dial it to bootstrap; the rest failed to join and exited, failing the sim. Set the bootstrap's cap to num_nodes+100 (matching the cluster bootstrap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
radiken
marked this pull request as ready for review
July 4, 2026 01:37
radiken
marked this pull request as draft
July 4, 2026 13:17
Adds four ExpConfig knobs rendered into shadow.yaml / node env: - seed: recorded explicitly in shadow.yaml (Shadow default 1) - model_unblocked_syscall_latency: escape hatch for zero-time spin loops - strace_logging_mode: global syscall logging for small-N diagnosis - lsquic_tick_floor_us: LSQUIC_TICK_FLOOR_US for the tick-floor node image, which fixes the quic livelock (lsquic diff=0 re-arm at one simulated instant) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lockstep process starts make every peer pair dial each other at the same simulated instant, forcing simultaneous-dial collisions that never occur on real hosts. pod-i now starts at 5000 + i*start_jitter_ms ms (default 0 keeps lockstep). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
radiken
marked this pull request as ready for review
July 14, 2026 17:56
AlbertoSoutullo
approved these changes
Jul 16, 2026
radiken
added a commit
that referenced
this pull request
Jul 16, 2026
Apply Pearson's review suggestion on #316: constrain muxer/discovery to their valid values with Literal instead of plain str, matching how nimlibp2p.py types them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
radiken
added a commit
that referenced
this pull request
Jul 16, 2026
Apply Pearson's review suggestion on #316: constrain muxer/discovery to their valid values with Literal instead of plain str, matching how nimlibp2p.py types them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
radiken
force-pushed
the
alan/shadow-regression-infra
branch
from
July 16, 2026 14:57
2d687e0 to
c7e3806
Compare
Constrain muxer/discovery to their valid values with Literal instead of plain str, matching how nimlibp2p.py types them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
radiken
force-pushed
the
alan/shadow-regression-infra
branch
from
July 16, 2026 15:00
c7e3806 to
6046e82
Compare
Pull the peer env and the peer / bootstrap / publisher host construction out of render_shadow_yaml into small pure functions so each is unit-testable. Output is unchanged; adds tests for the extracted builders. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PearsonWhite
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The shadow experiment can only run yamux in static mode currently, this PR adds muxer and discovery selection.
render_shadow_yamlgainsdiscoveryandstart_sleep.shadow-gossipsubExpConfig exposesmuxer,discovery,start_sleepand passes them through.multi-shadow-gossipsubis now env-configurable (SHADOW_MUXERS/SHADOW_SIZES/SHADOW_DISCOVERY).seed,strace_logging_mode,model_unblocked_syscall_latency,lsquic_tick_floor_us(needs the tick-floor node image),start_jitter_ms.Validated at 1000 nodes (kad-dht) against the cluster regression: mplex and yamux both hit 100% delivery. quic is now solved as well: the freeze was a zero-delay tick livelock in the lsquic wrapper (fixed by the tick-floor image +
lsquic_tick_floor_us) plus simultaneous-dial collisions from lockstep starts (fixed bystart_jitter_ms) - quic@1000 completes in ~1h25m at 100% delivery (600000/600000).