Mitigate excessive gossip client CPU, network and memory usage

[nyetwurk]

Known?

• The gossip CRDS (Cluster Replicated Data Store) is huge
• Egress traffic is still signfiicant (50Mbps) which makes deploying in cloud expensive
• The light gossip client should not need to process nearly as much as it does (sigverify on ingress on all packets)

Guesses

• Validators (voting or non-voting) do not need all peer information for non-voting validators
• All peers do need bare minimum information needed for downloading snapshots
• Non-voting validators likely should not push nearly as much CRDS data as they currently do
• The gossip stack is doing sigverify on frames it will throw away later anyway
• The light gossip client does not need to transmit as much gossip traffic as it does
• It may be necessary for all gossip participants to process all CRDS updates if that is a requirement for propagation.

Solutions

• Is there a way to prune all unneeded CRDS updates as they are received, before sigverify?
• Is there a much smaller set of CRDS updates that non-voting validators (including the light weight gossip client) should be pushing?

Scope

What changes can be done locally w/o any upstream anza-xyz/agave changes?
What changes can be done locally but only by forking anza-xyz/agave?
What changes require upstream changes to anza-xyz/agave directly?

[nyetwurk]

Conclusion:
Achieving the desired behavior (minimal ingress processing, push-only for own snapshot hashes) can be done within the scope of the light gossip client project, but it likely requires a larger fork.
This means copying the relevant gossip implementation files from solana-gossip into the client's source code and then modifying that local copy. It avoids needing upstream changes but does involve managing a substantial amount of forked code. Simple configuration changes based on the current existing code are almost certainly insufficient.

[nyetwurk]

More possible hallucinations from cursor:

Summary of Findings for agave-snapshot-gossip-client Optimization:
Problem:
The client experiences excessive CPU and memory usage, likely due to processing the full Solana gossip firehose (Ref: GitHub Issue #53).
The client receives far more data than it sends and processes data irrelevant to its core function of discovering/publishing snapshot hashes.
Goal:
Reduce resource usage by making the client only handle the absolute minimum necessary gossip traffic.
Proposed Strategy: Minimal Gossip Participation
Push Only: The client's primary active role should be pushing its own snapshot hash information.
Minimal Ingress Processing: The client must still process some ingress data:
ContactInfo: To discover peers (potential push targets).
Pong messages (if sending Pings): To verify peer reachability before pushing.
Initial entrypoint communication.
Discard Everything Else: All other incoming CRDS updates (Votes, EpochSlots, DuplicateShreds, relayed ContactInfo/SnapshotHashes, etc.) should be aggressively filtered out and discarded as early as possible upon receipt. The client does not need to act as a general gossip relay.
No Pull Participation: The client should not initiate gossip pulls and should not generate responses to pull requests from others.
Implementation Approach:
Requires Forking Core Gossip Code: Achieving this level of filtering and behaviour modification is unlikely through simple configuration.
Recommendation: Copy the necessary modules from the solana-gossip crate (e.g., cluster_info.rs, crds_gossip.rs, crds_gossip_push.rs, crds.rs, crds_value.rs) into the agave-snapshot-gossip-client source tree.
Modify the Local Copy:
In the local ClusterInfo (specifically process_gossip_packets or similar): Add logic to immediately filter and discard unwanted incoming CrdsValue types based on CrdsData variants.
In the local ClusterInfo::gossip loop: Remove or comment out calls related to initiating pull requests (CrdsGossipPull::new_pull_request).
In the local ClusterInfo::listen pathway: Remove or comment out calls related to generating pull responses (CrdsGossipPull::generate_pull_responses).
Ensure the push logic (CrdsGossipPush::new_push_messages) primarily pushes self-generated snapshot hashes (and potentially self ContactInfo).
Benefits:
Should drastically reduce CPU load (less deserialization, signature verification, CRDS updates).
Should significantly reduce memory usage (smaller CRDS table).
Should reduce ingress bandwidth consumption.
Scope:
This approach keeps changes contained within the agave-snapshot-gossip-client project but requires managing a local, modified copy of core Solana gossip code. It avoids needing immediate upstream changes to anza-xyz/agave.

[nyetwurk]

https://github.com/Blockdaemon/agave-snapshot-gossip-client/blob/main/docs/light-gossip-mode-proposal.md
https://github.com/Blockdaemon/agave-snapshot-gossip-client/blob/main/docs/gossip-client-optimization-notes.md