Having rek2 network issues, Longhorn won't complete the installation

[Daxcor69]

Hello,

I have installed the v1.22.7+rke2r2 using the install script. I am having trouble. The cluster looks like it installs correctly and I am able to issues kubectl commands against it. Now, when I go to install Longhorn, the installation hangs in a pending state. when I opened an issue on the longhorn github issues, they had seen this before and it comes down to a network issue in the cluster. No real details.

I am not sure how to diagnose this network issue. The logs seem to be cryptic.

Calico-node

2022-03-31 03:27:44.951 [INFO][62] status-reporter/watchercache.go 188: Failed to perform list of current data during resync ListRoot="/calico/resources/v3/projectcalico.org/caliconodestatuses" error=connection is unauthorized: caliconodestatuses.crd.projectcalico.org is forbidden: User "system:serviceaccount:kube-system:canal" cannot list resource "caliconodestatuses" in API group "crd.projectcalico.org" at the cluster scope
2022-03-31 03:27:45.952 [INFO][62] status-reporter/watchercache.go 175: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/caliconodestatuses"

This is a known issue, and will be fixed shortly apparently.

When I look at rke2-canal-xxx

Readiness probe failed: Get "http://localhost:9099/readiness": dial tcp [::1]:9099: connect: connection refused
Source kubelet worker1
Count 3
Sub-object spec.containers{calico-node}

In the Longorn pods..

Longhorn-driver-deployer

Failed to load logs: container "longhorn-driver-deployer" in pod "longhorn-driver-deployer-648cf57895-d5l4b" is waiting to start: PodInitializing
Reason: BadRequest (400)

I know there is more info that is required, but I am just not sure where to start. Any thoughts??

Brad

[brandond]

Can you link to the Longhorn issue you opened?

I believe Calico should currently be functional despite the error messages. Can you describe the pod to see why it's still initializing? Checking the containerd logs may also be helpful.

[Daxcor69]

longhorn/longhorn#3800

I am not sure any more.. feels like I am chasing my tail. rke2 and longhorn have be separate installs. I am thinking of trying Rancher to do the installation of the cluster and longhorn. I must be missing something in the requirements.

Brad

[brandond]

time="2022-03-30T08:33:52Z" level=debug msg="Failed to check for the latest and stable Longhorn versions" controller=longhorn-setting error="Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: read udp 10.42.1.23:50846->10.43.0.10:53: i/o timeout" node=worker1
time="2022-03-30T09:33:52Z" level=debug msg="Failed to check for the latest and stable Longhorn versions" controller=longhorn-setting error="Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: lookup longhorn-upgrade-responder.rancher.io on 10.43.0.10:53: read udp 10.42.1.23:50246->10.43.0.10:53: i/o timeout" node=worker1
time="2022-03-30T10:33:58Z" level=debug msg="Failed to check for the latest and stable Longhorn versions" controller=longhorn-setting error="Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout" node=worker1
time="2022-03-30T11:33:58Z" level=debug msg="Failed to check for the latest and stable Longhorn versions" controller=longhorn-setting error="Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout" node=worker1
time="2022-03-30T13:33:58Z" level=debug msg="Failed to check for the latest and stable Longhorn versions" controller=longhorn-setting error="Post \"https://longhorn-upgrade-responder.rancher.io/v1/checkupgrade\": dial tcp: i/o timeout" node=worker1

These messages would seem to indicate that the longhorn pods cant reach the cluster DNS server. Are all the other pods in the cluster running successfully? CoreDNS in particular? Are all the correct ports open between your nodes?

[Daxcor69]

You and I were talking in the slack channel earlier. Here is my firewall values..

Status: active

To                         Action      From
--                         ------      ----
Anywhere on tailscale0     ALLOW       Anywhere
41641/udp                  ALLOW       Anywhere
10.0.0.0/24 on eth1        ALLOW       Anywhere

All three of the nodes are on the 10.0.0.0/24 network. I can ping all of them successfully from each other.

There are two coredns pods running, the first one has this in the logs:

[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration MD5 = 7da3877dbcacfd983f39051ecafd33bd
CoreDNS-1.8.5
linux/amd64, go1.16.10b7, 8ddb6313
[ERROR] plugin/errors: 2 834383084245821452.8642406763911847984. HINFO: read udp 10.42.0.88:57967->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 834383084245821452.8642406763911847984. HINFO: read udp 10.42.0.88:35653->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 834383084245821452.8642406763911847984. HINFO: read udp 10.42.0.88:41347->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 834383084245821452.8642406763911847984. HINFO: read udp 10.42.0.88:56828->8.8.8.8:53: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"

The other:

.:53
[INFO] plugin/reload: Running configuration MD5 = 7da3877dbcacfd983f39051ecafd33bd
CoreDNS-1.8.5
linux/amd64, go1.16.10b7, 8ddb6313

eth0 is a public interface and eth1 is a private interface, 10.0.0.0/24

brad

[Daxcor69]

Oh crap, it was the firewall, I disabled it and it now all is showing green. This doesn't make sense. The only thing I can think of is the cluster is trying to talk over the public interface. I will have to work on getting this to talk on the private network only. sorry for the bother.

[brandond]

Most CNIs don't actually support running with firewalld/ufw enabled. You may be able to get it working if you also poke holes for both of the cluster CIDRs on the correct virtual interface for your CNI, but it's not something we really support so I won't be able to help with much.