[BUG][Ubuntu][Bare metal] OpenShift cluster becomes Unreachable

[dabcoder]

General information

• OS: Linux
• Hypervisor: QEMU / libvirt
• Did you run crc setup before starting it: Yes
• Running CRC on: Baremetal-Server

CRC version

CRC version: 2.20.0+f3a947

CRC status

DEBU CRC version: 2.20.0+f3a947
DEBU OpenShift version: 4.13.0
DEBU Podman version: 4.4.4
DEBU Running 'crc status'
CRC VM:          Running
OpenShift:       Unreachable (v4.13.0)
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     26.29GB
Cache Directory: /home/ubuntu/.crc/cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 12
- memory                                : 27648
- network-mode                          : user

Host Operating System

# Put the output of `cat /etc/os-release`
PRETTY_NAME="Ubuntu 22.04.2 LTS"

Steps to reproduce

1. Follow the instructions listed in https://github.com/crc-org/crc/wiki/Debugging-guide
2. crc ip returns 127.0.0.1
3. ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null core@127.0.0.1 returns core@127.0.0.1: Permission denied (publickey).

Expected

SSH into the VM should be possible. However, could it be due to the fact that crc ip is returning the IP address of the loopback interface, and should return a 192.168.x.x IP instead?

ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 74:56:3c:56:f1:1e brd ff:ff:ff:ff:ff:ff
    inet 162.19.88.183/24 metric 100 brd 162.19.88.255 scope global dynamic enp3s0f0
       valid_lft 62233sec preferred_lft 62233sec
    inet6 2001:41d0:306:2bb7::/56 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::7656:3cff:fe56:f11e/64 scope link
       valid_lft forever preferred_lft forever
4: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 74:56:3c:56:f1:1f brd ff:ff:ff:ff:ff:ff
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:d5:4d:5a brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

Worth noting that what prompted me to try to SSH into the VM is the fact that the OpenShift cluster is appearing as Unreachable (I've read through this previous issue by the way: #1888).

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

The cluster starts without visible issues, without needing to delete the existing install.

[cfergeau]

With user-mode networking (crc config get networking-mode), 127.0.0.1 is expected. In this configuration, the ssh port for the VM is remapped to 2222. Port 22 will try to connect to the host SSH daemon, not to the VM.

[dabcoder]

Thanks, added 2222 in the ssh config (which I was convinced I had previously done, but apparently not). I am now able to SSH in the VM. For the record:

crc config get network-mode
network-mode : user

[gbraad]

@cfergeau on a sidenote, does it make sense to add a documentation to describe the usage of this with an ssh config entry, and perhaps even set the correct key (or add configuration to use the user's ssh key?). WDYT?

[cfergeau]

This is already documented in https://github.com/crc-org/crc/wiki/Debugging-guide

[dabcoder]

Running into another issue now:

crc status
CRC VM:          Running
OpenShift:       Unreachable (v4.13.0)
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     26.29GB
Cache Directory: /home/ubuntu/.crc/cache

So the VM is Running, but I am getting a ssh: connect to host 127.0.0.1 port 2222: Connection refused error.
Verbose output:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/ubuntu/.ssh/config
debug1: /home/ubuntu/.ssh/config line 1: Applying options for crc
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 127.0.0.1 is address
debug3: ssh_connect_direct: entering
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: connect to address 127.0.0.1 port 2222: Connection refused
ssh: connect to host 127.0.0.1 port 2222: Connection refused

Here's the ssh config:

Host crc
    Hostname 127.0.0.1
    User core
    Port 2222
    IdentityFile ~/.crc/machines/crc/id_ecdsa
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null

sudo lsof -i:2222 returns no output. Have I missed anything here?

[gbraad]

It shows "unreachable" so something is afoot/weird; the status command itself uses SSH to check the disk usage. This returns 0, so it feels like something crashed inside the VM (or the daemon process). Can you check what the console says from the hypervisor management tools?

[gbraad]

https://github.com/crc-org/crc/wiki/Debugging-guide

This is wiki, not documentation.

[dabcoder]

It shows "unreachable" so something is afoot/weird; the status command itself uses SSH to check the disk usage. This returns 0, so it feels like something crashed inside the VM (or the daemon process). Can you check what the console says from the hypervisor management tools?

I am seeing frequent occurences of those logs lines produced by the libvirtd.service systemd unit:

Aug 22 13:01:18 ns3220936 libvirtd[226934]: 226934: error : virNetSocketReadWire:1793 : End of file while reading data: Input/output error

However, they also occur when the OpenShift cluster is in a Running state so I don't think this is the root cause of this issue.
Since we're using QEMU / libvirt on this host, is there anything else I could check?

[cfergeau]

sudo virsh list --all will show the state of the crc VM.
At this point, I'd try crc stop; crc start to see if it helps, or crc delete; crc start if you don't mind the cluster data loss.
Another thing to try is to kill the crc daemon, and stop/start crc.

[gbraad]

Is the crc daemon as process still active? Since if that disappeared, the VM becomes unreachable.