Skip to content

after reboot cilium hostfirewall could block playbook run #13335

Description

@blackluck

What happened?

If using cilium host firewall, so node level fw rules and block by default, then after node reboot could cilium after start could drop replies for working connections, see cilium/cilium#25448

This sometimes could cause playbook run to stuck. After reboot there comes some download tasks, some of them with shell modules which won't timeout if they miss the reply packet.

Not sure the exact requiremenst, we install cilium via helm values: kubeProxyReplacement: "true"
, hostFirewall.enabled: true and policyEnforcementMode: always

What did you expect to happen?

For at least a workaround give an option for wait after reboot before doing any other tasks. So if there anything on the nodes that needs to finish first than could wait for X seconds.

How can we reproduce it (as minimally and precisely as possible)?

Install cilium via helm with these values:

  • kubeProxyReplacement: "true"
  • hostFirewall.enabled: true
  • policyEnforcementMode: always

then create ciliumclusterwidenetworkpolicy to limit ssh access from ansible node (replace cidr) :

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: ssh-to-nodes
specs:
- ingress:
  - fromCIDR:
    - 10.1.2.3/32
    toPorts:
    - ports:
      - port: "22"
        protocol: TCP
  nodeSelector:
    matchLabels:
      reserved:host: ""

Then try to use kubespray upgrade with system_upgrade: true and system_upgrade_reboot: always to reboot nodes after system package upgrade.

OS

Ubuntu 24

Version of Ansible

created awx exexcution environment based on release 2.31.0 requirements
https://github.com/kubernetes-sigs/kubespray/blob/v2.31.0/requirements.txt

Version of Python

python3.11

Version of Kubespray (commit)

release-2.31.0

Network plugin used

cni

Full inventory with variables

not relevant

Command used to invoke ansible

awx

Output of ansible run

there isn't an error, had to cancel the run, because nothing happend even after hours
sometimes after some image donwload trying, like:

TASK [download : Download_container | Prepare container download] **************
included: /runner/project/roles/download/tasks/check_pull_required.yml for worker002

sometimes after other tusk, because it's based on what tasks try to run when cilium loads hostfirewall rules and if it blocks ansible traffic or not.

Anything else we need to know

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Ubuntu 24kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions