configuration details

[danehans]

Great job with kube-parrot. So far I really like what I see with the project. I am trying to setup parrot to work with my k8s cluster. Can you help me with the following questions?

• Does parrot run on kubelet nodes? If so, how do I configure the kubelet to use parrot? Do you have any example conf files that you can share?
• I see the gobgp lib is being used and parrot runs a grpc server. Can I use the gobgp client to intact with parrot?
• What is the version of Kubernetes that has been tested with parrot? Do any other prerequisites exist?
• If you have parrot running as a container and managed by systemd, do you have a sample systemd unit file you can share?

I have a parrot instance running in my k8s lab cluster and it has established bgp neighbor relationships with my two configured upstream bgp speakers. In general, I'm trying to figure out how parrot integrates with k8s and a few general grey areas. Any feedback would be greatly appreciated.

Here are a few details of my setup:

docker run -d --net=host -v /home/core/.kube/config:/etc/kube-parrot/kubeconfig sapcc/kube-parrot:v201611131942 --kubeconfig=/etc/kube-parrot/kubeconfig --as=64512 --local_address=10.30.118.155 --master_address=10.30.118.155 --service_subnet=10.30.118.160/28 --neighbor=10.30.118.138 --neighbor=10.30.118.139

$ docker ps | grep parr
118ec7171c84        sapcc/kube-parrot:v201611131942            "/parrot --kubeconfig"   46 hours ago        Up 46 hours                             thirsty_varahamihira

$ docker logs 118ec7171c84
Welcome to Kubernetes Parrot v201611131942
time="2017-01-18T07:24:45Z" level=info msg="Add a peer configuration for:10.30.118.138" Topic=Peer 
time="2017-01-18T07:24:45Z" level=info msg="Add a peer configuration for:10.30.118.139" Topic=Peer 
time="2017-01-18T07:25:28Z" level=info msg="Peer Up" Key=10.30.118.138 State="BGP_FSM_OPENCONFIRM" Topic=Peer 
time="2017-01-18T07:27:07Z" level=info msg="Peer Up" Key=10.30.118.139 State="BGP_FSM_OPENCONFIRM" Topic=Peer ```

[BugRoger]

Does parrot run on kubelet nodes? If so, how do I configure the kubelet to use parrot?

The intention of Parrot is that it runs out-of-band and speaks for the whole cluster. Unfortunately, this is currently not possible due to the missing implementation of add-path in go-bgp.

See: osrg/gobgp#1085

This means that it needs to run as a sidekick next to each kube-proxy and/or api-server in the cluster. Since it is announcing each proxy's hostIP and the apiserver master address. In our deployment we distinguish between masters and nodes. We run kube-proxy and kube-parrot on each master but not on nodes.

Do you have any example conf files that you can share?

We schedule Parrot as a static pod with a manifest like so:

apiVersion: v1
kind: Pod
metadata:
  name: kube-parrot
  namespace: kube-system
spec:
  hostNetwork: true
  volumes:
    - name: etc-kubernetes
      hostPath:
        path: /etc/kubernetes
  containers:
    - name: parrot
      image: sapcc/kube-parrot:v201611131942
      volumeMounts:
        - mountPath: /etc/kubernetes
          name: etc-kubernetes
          readOnly: true
      args:
        - --local_address={{.internal_network.address}}
        - --master_address={{.kubernetes.master_address}}
        - --service_subnet={{.kubernetes.services_subnet_cidr}}
        - --as={{.bgp.remote}}
        - --logtostderr
        - --v=5
        - --kubeconfig=/etc/kubernetes/config/parrot
{{range $neighbor := .bgp.neighbors -}}
        - --neighbor={{$neighbor.address}}
{{-end}}

Since it is also announcing and talking to the API servers (which is the master_address here) you need at least one Parrot co-located with an apiserver and configure it to connect on localhost. We use something like this at /etc/kubernetes/config/parrot.

apiVersion: v1
kind: Config
clusters:
  - name: local
    cluster:
      server: "http://localhost:8080"
contexts:
  - name: local 
    context:
      cluster: local
      user: local 
current-context: local
users:
  - name: local

I see the gobgp lib is being used and parrot runs a grpc server. Can I use the gobgp client to intact with parrot?

Yes, this is possible. The grpc server is started on port 12345.

See:
https://github.com/sapcc/kube-parrot/blob/master/pkg/bgp/server.go#L42-L45
https://github.com/sapcc/kube-parrot/blob/master/cmd/parrot/main.go#L45

What is the version of Kubernetes that has been tested with parrot? Do any other prerequisites exist?

We tested with 1.3.X and 1.4.6. It doesn't do any magic in regards to the Kubernetes API. Should work with anything 1.0+.

If you have parrot running as a container and managed by systemd, do you have a sample systemd unit file you can share?

We are using the kubelet as supervisor (https://kubernetes.io/docs/admin/static-pods/). The intention is to have a completely self-hosted Kubernetes cluster. We previously had a mix of systemd and Kubernetes supervision but that got old fast. 😄

I have a parrot instance running in my k8s lab cluster and it has established bgp neighbor relationships with my two configured upstream bgp speakers.

Awesome. I'm feeling bad not having had the time to publish a README and you had to reverse-engineer this. 🙁

If you want Parrot to announce your pod-subnets you need to label the nodes with an annotation parrot.sap.cc/podsubnet: 10.0.0.0/27. You might be able to do something smarter but in our setup we assign the subnets by convention and didn't have the time to implement our own Kubernetes cloud-provider that would handle this problem. We plan to do this with 1.6 when it's possible to write cloud-providers as plugins.

https://github.com/sapcc/kube-parrot/blob/master/pkg/controller/podsubnets.go#L55

[danehans]

@BugRoger thank you for the details. Now that I have a better understanding of parrot, I'll try getting it working with my k8s lab cluster. I'll keep you posted.

[danehans]

@BugRoger I finally got back to testing kube-parrot. I've made good progress thanks to your help. Parrot starts, creates the bgp neighbor relationships and advertises externalIPs as expected. However, the next-hop is my internal, non-routed network used for control traffic (apiserver<>kubelet, ipxe, etc..). Therefore, parrot advertises the externalIP routes to the ToR switches (BGP neighbors) with a next-hop of the node's internal interface IP instead of the node's external interface IP.

I've worked-around this issue by creating a static route on the ToR switches to the IP of each nodes internal interface IP with a next-hop of the node's external interface IP. I would like to avoid having to manage these routes and simply use the node's external interface IP.

From reviewing the parrot code, it appears the source of the issue is here. Even if I bind kube-proxy to the node's external interface IP, the Proxy.Status.HostIP still reports the node's internal interface IP.

Using a node's External IP is no help since it reports the node's internal IP for both InternalIP and ExternalIP. For example:

Name:			node1.example.com
Role:			
Labels:			beta.kubernetes.io/arch=amd64
			beta.kubernetes.io/os=linux
			kubernetes.io/hostname=node1.example.com
<SNIP>
Addresses:		10.10.10.35,10.10.10.35,node1.example.com
<SNIP>

Note: 10.10.10.x/24 is my internal, non-routed network used for mgt svcs.

Do you see this same behavior or do your nodes expose ExternalIP's for services through your internal network?

Do your nodes properly report the InternalIP and ExternalIP from Node.Status.Addresses?

In the meantime, I'm going to look into the k8s node controller code for registering nodes. I need to understand how the Internal/ExternalIP's are determined during the node registration process.