No curl response from pod service

[leonardas103]

Following the guide in k8s-on-raspbian/GUIDE.md.
When I run:

pi@raspberrypi2:~ $ curl -4 http://127.0.0.1:31118 -d "test"

The HTTP request is sent but it is forever awaiting a response.

Environment

• Docker version 19.03.6
• Kubernetes v1.17.3
• Operating System: Raspbian 10 (buster)
• Board: PI4

All pods are running

pi@raspberrypi2:~ $ kubectl get pods --all-namespaces
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
default       markdownrender-f9744b577-gsmk9         1/1     Running   0          14m
kube-system   coredns-6955765f44-hsbhx               1/1     Running   0          23m
kube-system   coredns-6955765f44-zzf9w               1/1     Running   0          23m
kube-system   etcd-raspberrypi2                      1/1     Running   0          24m
kube-system   kube-apiserver-raspberrypi2            1/1     Running   0          24m
kube-system   kube-controller-manager-raspberrypi2   1/1     Running   0          24m
kube-system   kube-proxy-9zx54                       1/1     Running   0          11m
kube-system   kube-proxy-st75l                       1/1     Running   0          23m
kube-system   kube-scheduler-raspberrypi2            1/1     Running   0          24m
kube-system   weave-net-qhmps                        2/2     Running   2          11m
kube-system   weave-net-vhq6j                        2/2     Running   1          16m

Node (raspberrypi3) successfully joins

pi@raspberrypi2:~ $ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
raspberrypi2   Ready    master   26m   v1.17.3
raspberrypi3   Ready    <none>   13m   v1.17.3

I'm using the right port:

pi@raspberrypi2:~ $ kubectl get service  markdownrender
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
markdownrender   NodePort   10.106.161.23   <none>        8080:31118/TCP   17m

Routes show the 31118 port open

pi@raspberrypi2:~ $ netstat -tulpn
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      
tcp        0      0 192.168.0.23:2379       0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      
tcp        0      0 192.168.0.23:2380       0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:2381          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:45009         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:6784          0.0.0.0:*               LISTEN      
tcp6       0      0 :::10250                :::*                    LISTEN      
tcp6       0      0 :::10251                :::*                    LISTEN      
tcp6       0      0 :::6443                 :::*                    LISTEN      
tcp6       0      0 :::10252                :::*                    LISTEN      
tcp6       2      0 :::31118                :::*                    LISTEN      
tcp6       0      0 :::10256                :::*                    LISTEN      
tcp6       0      0 :::22                   :::*                    LISTEN      
tcp6       0      0 :::6781                 :::*                    LISTEN      
tcp6       0      0 :::6782                 :::*                    LISTEN      
tcp6       0      0 :::6783                 :::*                    LISTEN      

Comments

I believe I am missing something very simple. I already tried reset and init and the problem persists.

[DwayneGit]

Just figured it out. You run the curl command from one of the nodes. If you did the same as me you ran it on master.

Edit:
It only worked from my first node. Still learning...
probably should be available from all nodes 🤷‍♂

[justinabrahms]

This is just a thoery, but I believe the issue here is that there's no external ip address associated with the service:

pi@raspberrypi2:~ $ kubectl get service  markdownrender
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
markdownrender   NodePort   10.106.161.23   <none>        8080:31118/TCP   17m

no external ip, only the cluster ip. You can access it only from the host it's scheduled on.

[DwayneGit]

It has to do with the iptables of the pis that are preventing them from communicating. They have to be using leagcy binaries these three commands solved my problem: sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT More on it here -> https://www.shogan.co.uk/kubernetes/building-a-raspberry-pi-kubernetes-cluster-part-1-routing/

…

On Wed, Mar 4, 2020, 12:40 AM Justin Abrahms ***@***.***> wrote: This is just a thoery, but I believe the issue here is that there's no external ip address associated with the service: ***@***.***:~ $ kubectl get service markdownrender NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE markdownrender NodePort 10.106.161.23 <none> 8080:31118/TCP 17m no external ip, only the cluster ip. You can access it only from the host it's scheduled on. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/teamserverless/k8s-on-raspbian/issues/24?email_source=notifications&email_token=AKQPGDTYPNOQM22GJZP2YQTRFXZXJA5CNFSM4K2WZLF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENWR4EQ#issuecomment-594353682>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKQPGDVAGLFJQJWRMY4WLTDRFXZXJANCNFSM4K2WZLFQ> .

[justinabrahms]

That website seems to suggest that those ip table rules are to allow the worker nodes to access the public internet via the primary node's internet connection. This was due to their odd networking configuration where the primary node served as a router for the worker nodes.

[DwayneGit]

You are correct @justinabrahms but they should all be on the sames network in kubernetes so any node should be able to hit a cluster ip, which wasn't happening here.
They should all be using one point of contact to the outside world through the master node. Doing this wont create a public ip but will make sure the master is holding that access point should you create one eventually.
From my understanding that is what kubernetes should be doing. Im still learning though...

[DwayneGit]

Wait... Those may be the wrong commands though.. One moment. Ill double check

[DwayneGit]

I'm sorry you are right those were the wrong commands. you want these:

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy 
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy 
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

Links:
"Kubernetes Raspberry Pi 4 Cluster"

[leonardas103]

@DwayneGit Thanks for your response. It seems that setting the iptables to legacy is critical for kubernetes with iptables-1.8.x.

Can someone please update the guide on k8s-on-raspbian/GUIDE.md to include this?