|
1 | | -# Cilium Reference |
| 1 | +## Cilium Reference |
2 | 2 |
|
3 | | -Helm chart **v1.19.4**. |
| 3 | +Helm chart **v1.19.5**. |
4 | 4 |
|
5 | 5 | --- |
6 | | - |
7 | | -## What it does |
8 | | - |
9 | | -Cilium replaces kube-proxy entirely (`kubeProxyReplacement: true`) and handles all pod networking via eBPF. It advertises LoadBalancer IPs to MikroTik via BGP and allocates IPs from the `10.0.45.0/24` pool via `CiliumLoadBalancerIPPool`. |
10 | | - |
11 | | -Gateway API ingress is handled by **envoy-gateway** (see `kubernetes/core/envoy-gateway/`). Cilium's built-in Gateway API controller and embedded Envoy are both disabled. |
12 | | - |
13 | | ---- |
14 | | - |
15 | | -## Key values |
16 | | - |
17 | | -### Routing |
18 | | - |
19 | | -| Value | Setting | Effect | |
20 | | -| ------------------------ | --------------- | ----------------------------------------------------------------------------------------- | |
21 | | -| `routingMode` | `native` | Pods route directly without encapsulation (VXLAN/Geneve off) | |
22 | | -| `autoDirectNodeRoutes` | `true` | Adds host routes for pod CIDRs on each node automatically | |
23 | | -| `ipv4NativeRoutingCIDR` | `10.244.0.0/16` | The pod CIDR — tells Cilium not to masquerade traffic within this range | |
24 | | -| `ipam.mode` | `kubernetes` | Cilium defers IP allocation to the Kubernetes node IPAM | |
25 | | -| `endpointRoutes.enabled` | `true` | Per-endpoint routes instead of per-node — more precise, needed for native routing | |
26 | | -| `enableIPv4BIGTCP` | `true` | Enables BIG TCP for IPv4 — allows GSO/GRO packets larger than 64KiB for better throughput | |
27 | | -| `pmtuDiscovery.enabled` | `true` | Path MTU discovery — avoids fragmentation on routes with smaller MTU | |
28 | | - |
29 | | -### kube-proxy replacement |
30 | | - |
31 | | -| Value | Setting | Effect | |
32 | | -| ------------------------------------- | --------------- | ------------------------------------------------------------------------------------------- | |
33 | | -| `kubeProxyReplacement` | `true` | Cilium handles all Service/ClusterIP traffic via eBPF, kube-proxy not needed | |
34 | | -| `k8sServiceHost` | `127.0.0.1` | API server address Cilium uses internally (local haproxy/LB) | |
35 | | -| `k8sServicePort` | `7445` | Port for the above | |
36 | | -| `kubeProxyReplacementHealthzBindAddr` | `0.0.0.0:10256` | Health endpoint on the same port kube-proxy would use — keeps node readiness checks working | |
37 | | - |
38 | | -### Load balancing |
39 | | - |
40 | | -| Value | Setting | Effect | |
41 | | -| ------------------------------- | ------------- | ------------------------------------------------------------------------------- | |
42 | | -| `loadBalancer.algorithm` | `maglev` | Consistent hashing for backend selection — same client always hits same pod | |
43 | | -| `loadBalancer.mode` | `dsr` | Direct Server Return mode for load balancer traffic | |
44 | | -| `loadBalancer.acceleration` | `best-effort` | Use XDP acceleration for LB when available, fall back gracefully | |
45 | | -| `localRedirectPolicies.enabled` | `true` | Enables local redirect policies for steering traffic to node-local backends | |
46 | | -| `socketLB.enabled` | `true` | Socket-level load balancing — intercepts at connect() for lower latency | |
47 | | -| `socketLB.hostNamespaceOnly` | `true` | Socket LB only in host namespace — avoids interference with pod networking | |
48 | | - |
49 | | -### BGP |
50 | | - |
51 | | -| Value | Setting | Effect | |
52 | | -| ------------------------- | ------- | ------------------------------------------------------------------------------------- | |
53 | | -| `bgpControlPlane.enabled` | `true` | Enables Cilium's BGP control plane (new API, replaces legacy `bgp.enabled`) | |
54 | | -| `l2announcements.enabled` | `false` | L2 announcements disabled — BGP is used exclusively for LoadBalancer IP advertisement | |
55 | | - |
56 | | -BGP sessions are configured via `bgp.yml` (`CiliumBGPClusterConfig` + `CiliumBGPPeerConfig` + `CiliumBGPAdvertisement`): |
57 | | - |
58 | | -- **Cilium ASN**: `64513` (all k8s nodes) |
59 | | -- **MikroTik ASN**: `64512` (peer: `10.0.10.1`) |
60 | | -- **Advertised IPs**: all `LoadBalancer` service IPs (from the `10.0.45.0/24` pool) |
61 | | - |
62 | | -MikroTik peer connections are managed in `terraform/mikrotik/bgp.tf`. |
63 | | - |
64 | | -> The `CiliumLoadBalancerIPPool` in `pools.yml` allocates `10.0.45.0/24` for LoadBalancer Services. |
65 | | -
|
66 | | -### Gateway API (disabled) |
67 | | - |
68 | | -| Value | Setting | Effect | |
69 | | -| -------------------- | ------- | ---------------------------------------------------- | |
70 | | -| `gatewayAPI.enabled` | `false` | Cilium's built-in Gateway API controller is disabled | |
71 | | -| `envoy.enabled` | `false` | Cilium's embedded Envoy proxy is disabled | |
72 | | - |
73 | | -Gateway API is now managed by envoy-gateway running in the `networking` namespace. See `kubernetes/core/envoy-gateway/reference.md` for details. Cilium still provides the LoadBalancer IP allocation (via `CiliumLoadBalancerIPPool`) and BGP advertisement for the envoy-gateway Services. |
74 | | - |
75 | | -### BPF / eBPF |
76 | | - |
77 | | -| Value | Setting | Effect | |
78 | | -| ---------------------- | -------- | -------------------------------------------------------------------------- | |
79 | | -| `bpf.datapathMode` | `netkit` | Uses netkit device mode — newer, faster alternative to veth pairs | |
80 | | -| `bpf.masquerade` | `true` | eBPF-based masquerade instead of iptables for SNAT | |
81 | | -| `bpf.preallocateMaps` | `true` | Pre-allocates BPF map memory at startup — avoids runtime allocation stalls | |
82 | | -| `bpf.lbModeAnnotation` | `true` | Allows per-service LB mode override via annotation | |
83 | | - |
84 | | -### cgroup |
85 | | - |
86 | | -| Value | Setting | Effect | |
87 | | -| -------------------------- | ---------------- | ------------------------------------------------------------------ | |
88 | | -| `cgroup.autoMount.enabled` | `false` | Cilium does not mount cgroupv2 — already mounted by the OS (Talos) | |
89 | | -| `cgroup.hostRoot` | `/sys/fs/cgroup` | Path to the host cgroup filesystem | |
90 | | - |
91 | | -### Observability |
92 | | - |
93 | | -| Value | Setting | Effect | |
94 | | -| ------------------------------------------------ | ------------ | ----------------------------------------------------------- | |
95 | | -| `dashboards.enabled` | `true` | Deploy Grafana dashboard ConfigMaps for Cilium metrics | |
96 | | -| `dashboards.annotations.grafana_folder` | `Kubernetes` | Place dashboards in the "Kubernetes" folder in Grafana | |
97 | | -| `operator.dashboards.enabled` | `true` | Deploy Grafana dashboard ConfigMaps for the Cilium operator | |
98 | | -| `operator.dashboards.annotations.grafana_folder` | `Kubernetes` | Same folder for operator dashboards | |
99 | | - |
100 | | -### Misc |
101 | | - |
102 | | -| Value | Setting | Effect | |
103 | | -| ---------------------- | ------- | --------------------------------------------------------------- | |
104 | | -| `cni.exclusive` | `false` | Allow other CNI plugins alongside Cilium (needed for Multus) | |
105 | | -| `hubble.enabled` | `true` | Hubble observability plane enabled | |
106 | | -| `operator.rollOutPods` | `true` | Operator pods restart automatically on ConfigMap/Secret changes | |
107 | | -| `rollOutCiliumPods` | `true` | Agent pods restart automatically on config changes | |
108 | | - |
109 | | ---- |
110 | | - |
111 | | -## Security capabilities |
112 | | - |
113 | | -The `securityContext.capabilities` block is required for the eBPF agent to function — `NET_ADMIN`, `SYS_ADMIN`, `NET_RAW` etc. are all needed for loading BPF programs and managing network interfaces. `cleanCiliumState` runs on uninstall/restart to tear down BPF maps. |
114 | | - |
115 | | ---- |
116 | | - |
117 | | -## Troubleshooting |
118 | | - |
119 | | -```bash |
120 | | -# Check Cilium agent status on a node |
121 | | -kubectl -n kube-system exec ds/cilium -- cilium status |
122 | | - |
123 | | -# Check BGP peer session state |
124 | | -kubectl -n kube-system exec ds/cilium -- cilium bgp peers |
125 | | - |
126 | | -# Check BGP route advertisements |
127 | | -kubectl -n kube-system exec ds/cilium -- cilium bgp routes |
128 | | - |
129 | | -# Check LoadBalancer IP pool usage |
130 | | -kubectl get ciliumloadbalancerippools |
131 | | - |
132 | | -# Verify kube-proxy replacement |
133 | | -kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxy |
134 | | - |
135 | | -# On MikroTik: verify BGP sessions and installed routes |
136 | | -# /routing bgp session print |
137 | | -# /ip route print where bgp |
138 | | -``` |
0 commit comments