updating IP_COOLDOWN_PERIOD from 30 seconds to 130 seconds to ensure SYN_SENT (120 seconds) conntrack timer expires #3636
Open
Mohijeet wants to merge 1 commit intoaws:masterfrom
Open
updating IP_COOLDOWN_PERIOD from 30 seconds to 130 seconds to ensure SYN_SENT (120 seconds) conntrack timer expires #3636Mohijeet wants to merge 1 commit intoaws:masterfrom
Mohijeet wants to merge 1 commit intoaws:masterfrom
Conversation
Contributor
|
@Mohijeet, thanks for the contribution. But increasing the default to 130s would significantly reduce IP churn rate in high-pod-turnover clusters, potentially causing IP starvation issues. I don't think we should set 130s as default value. |
|
What is the recommended fix for this behavior? Should the VPC CNI explicitly configure the SYN_SENT timeout or retry parameters during startup to override the default Linux kernel settings? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
improvement
Which issue does this PR fix?:
In-flight TCP connections to a deleted pod's IP can be silently hijacked if a new pod is assigned the same IP within the conntrack SYN_SENT expiry window (120 seconds on most Linux kernels).
issue
What does this PR do / Why do we need it?:
Problem:
This is dangerous because:
Fix:
Increase the default IP cooldown period from 30 seconds to 130 seconds. This ensures:
All SYN_SENT conntrack entries from the deleted pod's IP have fully expired (120 seconds on most Linux OS)
All tcp_syn_retries for in-flight connections have been exhausted (up to 130 seconds)
Only two code changes are made:
Default value in getCooldownPeriod() changed from 30 to 130
Updated the envIPCooldownPeriod constant comment to document the new default and reasoning
The IP_COOLDOWN_PERIOD environment variable continues to work as before, allowing operators to override this value if needed.
Will this PR introduce any new dependencies?:
No.
Will this break upgrades or downgrades? Has updating a running cluster been tested?:
No breaking changes. The cooldown period increase is a conservative safety improvement. On upgrade, existing clusters will automatically benefit from the longer cooldown. Operators who need a shorter cooldown can override via the IP_COOLDOWN_PERIOD environment variable.
Does this change require updates to the CNI daemonset config files to work?:
No. Works with a kubectl patch of the image tag. No config changes required.
Does this PR introduce any user-facing change?:
Yes. The default IP cooldown period after pod deletion is increased from 30 seconds to 130 seconds. This may slightly reduce IP churn rate in high-pod-turnover clusters but prevents silent TCP connection hijacking via stale conntrack entries.