What happened?
If using cilium host firewall, so node level fw rules and block by default, then after node reboot could cilium after start could drop replies for working connections, see cilium/cilium#25448
This sometimes could cause playbook run to stuck. After reboot there comes some download tasks, some of them with shell modules which won't timeout if they miss the reply packet.
Not sure the exact requiremenst, we install cilium via helm values: kubeProxyReplacement: "true"
, hostFirewall.enabled: true and policyEnforcementMode: always
What did you expect to happen?
For at least a workaround give an option for wait after reboot before doing any other tasks. So if there anything on the nodes that needs to finish first than could wait for X seconds.
How can we reproduce it (as minimally and precisely as possible)?
Install cilium via helm with these values:
- kubeProxyReplacement: "true"
- hostFirewall.enabled: true
- policyEnforcementMode: always
then create ciliumclusterwidenetworkpolicy to limit ssh access from ansible node (replace cidr) :
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: ssh-to-nodes
specs:
- ingress:
- fromCIDR:
- 10.1.2.3/32
toPorts:
- ports:
- port: "22"
protocol: TCP
nodeSelector:
matchLabels:
reserved:host: ""
Then try to use kubespray upgrade with system_upgrade: true and system_upgrade_reboot: always to reboot nodes after system package upgrade.
OS
Ubuntu 24
Version of Ansible
created awx exexcution environment based on release 2.31.0 requirements
https://github.com/kubernetes-sigs/kubespray/blob/v2.31.0/requirements.txt
Version of Python
python3.11
Version of Kubespray (commit)
release-2.31.0
Network plugin used
cni
Full inventory with variables
not relevant
Command used to invoke ansible
awx
Output of ansible run
there isn't an error, had to cancel the run, because nothing happend even after hours
sometimes after some image donwload trying, like:
TASK [download : Download_container | Prepare container download] **************
included: /runner/project/roles/download/tasks/check_pull_required.yml for worker002
sometimes after other tusk, because it's based on what tasks try to run when cilium loads hostfirewall rules and if it blocks ansible traffic or not.
Anything else we need to know
No response
What happened?
If using cilium host firewall, so node level fw rules and block by default, then after node reboot could cilium after start could drop replies for working connections, see cilium/cilium#25448
This sometimes could cause playbook run to stuck. After reboot there comes some download tasks, some of them with shell modules which won't timeout if they miss the reply packet.
Not sure the exact requiremenst, we install cilium via helm values: kubeProxyReplacement: "true"
, hostFirewall.enabled: true and policyEnforcementMode: always
What did you expect to happen?
For at least a workaround give an option for wait after reboot before doing any other tasks. So if there anything on the nodes that needs to finish first than could wait for X seconds.
How can we reproduce it (as minimally and precisely as possible)?
Install cilium via helm with these values:
then create ciliumclusterwidenetworkpolicy to limit ssh access from ansible node (replace cidr) :
Then try to use kubespray upgrade with
system_upgrade: trueandsystem_upgrade_reboot: alwaysto reboot nodes after system package upgrade.OS
Ubuntu 24
Version of Ansible
created awx exexcution environment based on release 2.31.0 requirements
https://github.com/kubernetes-sigs/kubespray/blob/v2.31.0/requirements.txt
Version of Python
python3.11
Version of Kubespray (commit)
release-2.31.0
Network plugin used
cni
Full inventory with variables
not relevant
Command used to invoke ansible
awx
Output of ansible run
there isn't an error, had to cancel the run, because nothing happend even after hours
sometimes after some image donwload trying, like:
sometimes after other tusk, because it's based on what tasks try to run when cilium loads hostfirewall rules and if it blocks ansible traffic or not.
Anything else we need to know
No response