[BUG] Newer version of traefik doesn't work with k3d

[rohitsakala]

What did you do

• How was the cluster created?

  • k3d cluster create --k3s-server-arg --disable --k3s-server-arg traefik

• What did you do afterwards?

• Install epinio which installs traefik 2.3.3 as one of the components.

What did you expect to happen

epinio creates an ingress with ingress.class=traefik and uses a tls secret which has a self-sgined certificate from cert-manager component installed by epinio.

So, when I do a curl to https://epinio.192.168.99.3.omg.howdoi.website from outside the cluster, it works.
When I do a curl after exec into a pod (inside the kube cluster), it fails with this error

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to epinio-registry.172.18.0.3.omg.howdoi.website:443 

When I run k3d cluster create, which uses the old version of traefik 1.7.19, it works perfectly. But, when I use the latest version of traefik 2.3.3, it fails. Also, all versions of traefik work fine in minikube and kind.

Screenshots or terminal output

I tried using openssl s_client and I get the following output which is not so helpful for me

epinio-server-6c96cc65b7-v2bjn:/ # openssl s_client -connect epinio-registry.192.168.128.3.omg.howdoi.website:443 -state -msg -debug
CONNECTED(00000003)
SSL_connect:before SSL initialization
>>> ??? [length 0005]
    16 03 01 01 59
>>> TLS 1.3, Handshake [length 0159], ClientHello
    01 00 01 55 03 03 ae aa 06 8a f8 74 94 d0 7a 2a
    57 31 6b a3 df e4 0e f5 c8 f6 45 86 da 09 f6 af
    19 56 a7 47 32 c5 20 e6 e5 04 75 b6 29 6d b6 36
    a8 22 fe 80 ae 97 c5 d1 38 ec 74 9d 17 64 58 bc
    3b 48 1c a9 82 27 36 00 3e 13 02 13 03 13 01 c0
    2c c0 30 00 9f cc a9 cc a8 cc aa c0 2b c0 2f 00
    9e c0 24 c0 28 00 6b c0 23 c0 27 00 67 c0 0a c0
    14 00 39 c0 09 c0 13 00 33 00 9d 00 9c 00 3d 00
    3c 00 35 00 2f 00 ff 01 00 00 ce 00 00 00 35 00
    33 00 00 30 65 70 69 6e 69 6f 2d 72 65 67 69 73
    74 72 79 2e 31 39 32 2e 31 36 38 2e 31 32 38 2e
    33 2e 6f 6d 67 2e 68 6f 77 64 6f 69 2e 77 65 62
    73 69 74 65 00 0b 00 04 03 00 01 02 00 0a 00 0c
    00 0a 00 1d 00 17 00 1e 00 19 00 18 00 23 00 00
    00 16 00 00 00 17 00 00 00 0d 00 30 00 2e 04 03
    05 03 06 03 08 07 08 08 08 09 08 0a 08 0b 08 04
    08 05 08 06 04 01 05 01 06 01 03 03 02 03 03 01
    02 01 03 02 02 02 04 02 05 02 06 02 00 2b 00 09
    08 03 04 03 03 03 02 03 01 00 2d 00 02 01 01 00
    33 00 26 00 24 00 1d 00 20 bb 8a 19 95 eb 44 d5
    57 f3 bd 11 9a f7 46 e5 78 85 57 65 7f 25 dd 8c
    3d 8b 1e 2e 6b 04 f2 0b 5c
write to 0x5580b22fd310 [0x5580b231d990] (350 bytes => 350 (0x15E))
0000 - 16 03 01 01 59 01 00 01-55 03 03 ae aa 06 8a f8   ....Y...U.......
0010 - 74 94 d0 7a 2a 57 31 6b-a3 df e4 0e f5 c8 f6 45   t..z*W1k.......E
0020 - 86 da 09 f6 af 19 56 a7-47 32 c5 20 e6 e5 04 75   ......V.G2. ...u
0030 - b6 29 6d b6 36 a8 22 fe-80 ae 97 c5 d1 38 ec 74   .)m.6."......8.t
0040 - 9d 17 64 58 bc 3b 48 1c-a9 82 27 36 00 3e 13 02   ..dX.;H...'6.>..
0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa   .....,.0........
0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27   .+./...$.(.k.#.'
0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d   .g.....9.....3..
0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 ce   ...=.<.5./......
0090 - 00 00 00 35 00 33 00 00-30 65 70 69 6e 69 6f 2d   ...5.3..0epinio-
00a0 - 72 65 67 69 73 74 72 79-2e 31 39 32 2e 31 36 38   registry.192.168
00b0 - 2e 31 32 38 2e 33 2e 6f-6d 67 2e 68 6f 77 64 6f   .128.3.omg.howdo
00c0 - 69 2e 77 65 62 73 69 74-65 00 0b 00 04 03 00 01   i.website.......
00d0 - 02 00 0a 00 0c 00 0a 00-1d 00 17 00 1e 00 19 00   ................
00e0 - 18 00 23 00 00 00 16 00-00 00 17 00 00 00 0d 00   ..#.............
00f0 - 30 00 2e 04 03 05 03 06-03 08 07 08 08 08 09 08   0...............
0100 - 0a 08 0b 08 04 08 05 08-06 04 01 05 01 06 01 03   ................
0110 - 03 02 03 03 01 02 01 03-02 02 02 04 02 05 02 06   ................
0120 - 02 00 2b 00 09 08 03 04-03 03 03 02 03 01 00 2d   ..+............-
0130 - 00 02 01 01 00 33 00 26-00 24 00 1d 00 20 bb 8a   .....3.&.$... ..
0140 - 19 95 eb 44 d5 57 f3 bd-11 9a f7 46 e5 78 85 57   ...D.W.....F.x.W
0150 - 65 7f 25 dd 8c 3d 8b 1e-2e 6b 04 f2 0b 5c         e.%..=...k...\
SSL_connect:SSLv3/TLS write client hello
read from 0x5580b22fd310 [0x5580b2314773] (5 bytes => 0 (0x0))
SSL_connect:error in SSLv3/TLS write client hello
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 350 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x5580b22fd310 [0x5580b223f700] (8192 bytes => 0 (0x0))

Which OS & Architecture

• Linux. OpenSUSE Leap 15.2

Which version of k3d

k3d version v4.4.4
k3s version v1.20.6-k3s1 (default)

Which version of docker

• docker version

Client:
 Version:           19.03.15
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        99e3ed89195c
 Built:             Sun Feb 14 12:00:00 2021
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.15
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       99e3ed89195c
  Built:            Sun Feb 14 12:00:00 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.9
  GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.1.5_catatonit
  GitCommit:

• docker info

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 19.03.15
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc oci
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ea765aba0d05254012b0b9e595e995c09186427f
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.18-24.52-default
 Operating System: SUSE Linux Enterprise Server 15 SP2
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 62.53GiB
 Name: 0100164092184
 ID: P3VB:RYHK:FGEW:CFYA:VQR3:ZMEC:RUXA:MJMT:6A6H:IL7Z:TOBR:XH5Y
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[iwilltry42]

Hi @rohitsakala , thanks for starting this discussion!
So you want to access your TLS secured ingress from inside the k3d/k3s cluster, within a pod running in that same cluster?
Since you're executing that inside a pod, I'm not sure, how the runtime might be involved there 🤔
One thing that may be worth noting is, that the official k3s image is not based on an OS (unlike e.g. the kind base image IIRC), so it only includes the certificate bundles baked in at built-time: https://github.com/k3s-io/k3s/blob/master/package/Dockerfile#L6 .
Maybe copying the certs from your host into the pod or just the k3d nodes could help 🤔

[rohitsakala]

Hello, I tried copying the ca cert and trying the curl but it still didn't work.

Anyways, we found out the problem. The problem is that linkerd side car and klipper-lb were both trying to change iptables and it messed up the routing.

[iwilltry42]

Ah so it's unrelated to traefik itself 👍
You can disable klipper already at cluster creation time using --k3s-server-arg "--disable=servicelb".
Or afterwards just delete it from the cluster.

[rohitsakala]

Gotcha. Thanks for the reply.