Skip to content

Conversation

@brandond
Copy link
Member

@brandond brandond commented Apr 16, 2025

Proposed Changes

  • Add cache to spegel p2p peerinfo handler
    libp2p may make a large number of bootstrap calls during startup; serve nodes from cache to avoid excessive CPU usage.
  • Move delegating auth middleware into common package and add MaxInFlight
    Adds maximum in-flight request limits to agent join and p2p peer info request request handlers.
  • Fix chainingBootstrapper to return the first successful address list
    Avoids infinite recursion when the chain includes an agentBootstrapper with a server address that points back at this node (via join address loop or external LB)

Types of Changes

bugfix

Verification

See linked issue: #12127 (comment)

  1. Start server1 with cluster-init: true.
  2. Join server2 against server1.
  3. Join server3 against server2.
  4. Remove cluster-init: true from server1 , and add server: https://server3:6443 instead.
  5. Do kubectl get --raw /v1-k3s/p2p on any one of the servers.
  6. Note that the servers are now stuck in a loop requesting addresses from each other. This doesn't affect availability, but it does eat up a fair bit of CPU time. If debug logging is enabled, you can see a stream of Serving p2p peer addrs message in the logs.

Testing

Linked Issues

User-Facing Change

Further Comments

@brandond brandond requested a review from a team as a code owner April 16, 2025 22:10
@brandond brandond changed the title Fix excessive load when Fix excessive load when bootstrapping P2P peers Apr 16, 2025
@codecov
Copy link

codecov bot commented Apr 16, 2025

Codecov Report

Attention: Patch coverage is 28.98551% with 49 lines in your changes missing coverage. Please review.

Project coverage is 45.20%. Comparing base (d824d5d) to head (a69a65f).
Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
pkg/server/auth/auth.go 19.51% 33 Missing ⚠️
pkg/spegel/bootstrap.go 0.00% 8 Missing ⚠️
pkg/cluster/https.go 78.57% 2 Missing and 1 partial ⚠️
pkg/spegel/spegel.go 0.00% 3 Missing ⚠️
pkg/agent/https/https.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #12154      +/-   ##
==========================================
+ Coverage   45.13%   45.20%   +0.06%     
==========================================
  Files         189      188       -1     
  Lines       19212    19210       -2     
==========================================
+ Hits         8672     8683      +11     
+ Misses       9305     9294      -11     
+ Partials     1235     1233       -2     
Flag Coverage Δ
e2etests 35.70% <28.98%> (+0.06%) ⬆️
inttests 34.79% <28.98%> (-0.01%) ⬇️
unittests 19.82% <13.04%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@brandond brandond changed the title Fix excessive load when bootstrapping P2P peers Fix excessive load when bootstrapping P2P peers from servers configured in loop Apr 16, 2025
@brandond brandond changed the title Fix excessive load when bootstrapping P2P peers from servers configured in loop Fix excessive load when bootstrapping P2P peers from servers configured in a join loop Apr 16, 2025
dereknola
dereknola previously approved these changes Apr 16, 2025
libp2p may make a large number of bootstrap calls during startup; serve nodes from cache to avoid excessive CPU usage.

Signed-off-by: Brad Davidson <[email protected]>
Adds maximum in-flight request limits to agent join and p2p peer info
request request handlers.

Signed-off-by: Brad Davidson <[email protected]>
Avoids infinite recursion when the chain includes an agentBootstrapper with a server address that points back at this node (via join address loop or external LB)

Signed-off-by: Brad Davidson <[email protected]>
@brandond brandond force-pushed the fix-spegel-bootstrap-recursion branch from 294176c to a69a65f Compare April 17, 2025 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants