Skip to content

Commit aad57ed

Browse files
mohamed-essammohamed-essam
committed
WIP
1 parent dac4217 commit aad57ed

File tree

13 files changed

+247
-39
lines changed

13 files changed

+247
-39
lines changed

‎README.md‎

Lines changed: 68 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,46 @@
11
# NetBird Kubernetes Operator
22
For easily provisioning access to Kubernetes resources using NetBird.
33

4+
https://github.com/user-attachments/assets/5472a499-e63d-4301-a513-ad84cfe5ca7b
5+
46
## Description
57

68
This operator enables easily provisioning NetBird access on kubernetes clusters, allowing users to access internal resources directly.
79

810
## Getting Started
911

1012
### Prerequisites
11-
- helm version 3+
13+
- (Recommended) helm version 3+
1214
- kubectl version v1.11.3+.
1315
- Access to a Kubernetes v1.11.3+ cluster.
14-
- (Optional for Helm chart installation) Cert Manager.
15-
16-
### To Deploy on the cluster
16+
- (Recommended) Cert Manager.
1717

18-
**Using the install.yaml**
1918

20-
```sh
21-
kubectl create namespace netbird
22-
kubectl apply -n netbird -f https://github.com/netbirdio/kubernetes-operator/releases/latest/manifests/install.yaml
23-
```
19+
### Deployment
20+
> [!NOTE]
21+
> Helm Installation method is recommended due to automation of multiple settings within the deployment.
2422
25-
**Using the Helm Chart**
23+
#### Using Helm
2624

25+
1. Add helm repository.
2726
```sh
2827
helm repo add netbirdio https://netbirdio.github.io/kubernetes-operator
29-
helm install -n netbird netbird-operator netbirdio/netbird-operator
3028
```
29+
2. (Recommended) Install [cert-manager](https://cert-manager.io/docs/installation/#default-static-install).
30+
1. (Recommended) Create a values.yaml file, check `helm show values netbirdio/kubernetes-operator` for more info.
31+
1. Install using `helm install --create-namespace -f values.yaml -n netbird netbird-operator netbirdio/kubernetes-operator`.
3132

32-
For more options, check the default values by running
33-
```sh
34-
helm show values netbirdio/netbird-operator
35-
```
33+
#### Using install.yaml
3634

37-
### To Uninstall
38-
**Using install.yaml**
35+
> [!IMPORTANT]
36+
> install.yaml only includes a very basic template for deploying a stripped down version of kubernetes-operator.
37+
> This excludes any and all configuration for ingress capabilities, and requires cert-manager to be installed.
3938
4039
```sh
41-
kubectl delete -n netbird -f https://github.com/netbirdio/kubernetes-operator/releases/latest/manifests/install.yaml
42-
kubectl delete namespace netbird
40+
kubectl create namespace netbird
41+
kubectl apply -n netbird -f https://raw.githubusercontent.com/netbirdio/kubernetes-operator/refs/heads/main/manifests/install.yaml
4342
```
4443

45-
**Using helm**
46-
47-
```sh
48-
helm uninstall -n netbird netbird-operator
49-
```
5044

5145
### Provision pods with NetBird access
5246

@@ -94,8 +88,58 @@ spec:
9488
containers:
9589
- image: yourimage
9690
name: container
91+
```
92+
93+
## Provisioning Networks (Ingress Functionality)
94+
95+
### Granting controller access to NetBird Management
96+
97+
> [!IMPORTANT]
98+
> NetBird kubernetes operator generates configurations using NetBird API, editing or deleting these configurations in the NetBird console may cause temporary network disconnection until the operator reconciles the configuration.
99+
100+
1. Create a Service User on your NetBird dashboard (Must be Admin). [Doc](https://docs.netbird.io/how-to/access-netbird-public-api#creating-a-service-user).
101+
1. Create access token for the Service User (Must be Admin). [Doc](https://docs.netbird.io/how-to/access-netbird-public-api#creating-a-service-user).
102+
1. Add access token to your helm values file under `netbirdAPI.key`.
103+
1. Alternatively, provision secret in the same namespace as the operator and set the key `NB_API_KEY` to the access token generated.
104+
1. Set `netbirdAPI.keyFromSecret` to the name of the secret created.
105+
1. Set `ingress.enabled` to `true`.
106+
1. Optionally, to provision network immediately, set `ingress.router.enabled` to `true`.
107+
1. Optionally, to provision 1 network per kubernetes namespace, set `ingress.namespacedNetworks` to `true`.
108+
1. Run `helm install` or `helm upgrade`.
97109

110+
### Exposing a Service
111+
112+
> [!IMPORTANT]
113+
> Ingress DNS Resolution requires DNS Wildcard Routing to be enabled, and at least one DNS Nameserver configured for clients.
114+
115+
|Annotation|Description|Default|
116+
|---|---|---|
117+
|`netbird.io/expose`|Expose service using NetBird Network Resource||
118+
|`netbird.io/groups`|Comma-separated list of group names to assign to Network Resource|`{ClusterName}-{Namespace}-{Service}`|
119+
|`netbird.io/resource-name`|Network Resource name|`{Namespace}-{Service}`|
120+
|`netbird.io/policy`|Name of NBPolicy to propagate service ports as destination.||
121+
|`netbird.io/policy-ports`|Narrow down exposed ports in policy, leave empty for all ports.||
122+
|`netbird.io/policy-protocol`|Narrow down protocol for use in policy, leave empty for all protocols.||
123+
124+
### Managing Policies
125+
126+
1. Simply add policies under `ingress.policies`, for example.
127+
```yaml
128+
ingress:
129+
policies:
130+
default:
131+
name: Kubernetes Default Policy # Required
132+
description: Default # Optional
133+
sourceGroups: # Required
134+
- All
135+
ports: # Optional, resources annotated 'netbird.io/policy=default' will append to this
136+
- 443
137+
protocols: # Optional, restricts protocols allowed to resources, defaults to ['tcp', 'udp']
138+
- tcp
139+
- udp
140+
bidirectional: true # Optional, defaults to true
98141
```
142+
2. Reference policy in Services using `netbird.io/policy=default`, this will add relevant ports and destination groups to Policy.
99143

100144
## Contributing
101145

‎examples/ingress/exposed-nginx.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: test
6+
name: test
7+
namespace: default
8+
spec:
9+
replicas: 1
10+
selector:
11+
matchLabels:
12+
app: test
13+
strategy:
14+
rollingUpdate:
15+
maxSurge: 25%
16+
maxUnavailable: 25%
17+
type: RollingUpdate
18+
template:
19+
metadata:
20+
labels:
21+
app: test
22+
spec:
23+
containers:
24+
- image: nginx
25+
imagePullPolicy: Always
26+
name: nginx
27+
---
28+
apiVersion: v1
29+
kind: Service
30+
metadata:
31+
annotations:
32+
netbird.io/expose: "true"
33+
netbird.io/policy: default
34+
netbird.io/resource-name: nginx
35+
labels:
36+
app: test
37+
name: test
38+
namespace: default
39+
spec:
40+
ports:
41+
- name: http
42+
port: 80
43+
protocol: TCP
44+
targetPort: 80
45+
selector:
46+
app: test
47+
type: ClusterIP

‎examples/ingress/values.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
ingress:
2+
enabled: true
3+
router:
4+
enabled: true
5+
policies:
6+
default:
7+
name: Kubernetes Default Policy
8+
sourceGroups:
9+
- All
10+
11+
netbirdAPI:
12+
key: "nbp_m0LM9ZZvDUzFO0pY50iChDOTxJgKFM3DIqmZ" # Replace with valid NetBird Service Account token

‎examples/setup-keys/example.yaml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: test
5+
namespace: default
6+
stringData:
7+
SETUP_KEY: 50445ABC-8901-4050-8047-0A390658A79B # Replace with valid setup key
8+
---
9+
apiVersion: netbird.io/v1
10+
kind: NBSetupKey
11+
metadata:
12+
name: test
13+
namespace: default
14+
spec:
15+
secretKeyRef:
16+
name: test
17+
key: SETUP_KEY
18+
---
19+
apiVersion: apps/v1
20+
kind: Deployment
21+
metadata:
22+
labels:
23+
app: test
24+
name: test
25+
namespace: default
26+
spec:
27+
replicas: 1
28+
selector:
29+
matchLabels:
30+
app: test
31+
strategy:
32+
rollingUpdate:
33+
maxSurge: 25%
34+
maxUnavailable: 25%
35+
type: RollingUpdate
36+
template:
37+
metadata:
38+
labels:
39+
app: test
40+
annotations:
41+
netbird.io/setup-key: test
42+
spec:
43+
containers:
44+
- image: ubuntu
45+
imagePullPolicy: Always
46+
name: ubuntu
47+
command:
48+
- sleep
49+
- inf

‎helm/netbird-operator/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@ apiVersion: v2
22
name: netbird-operator
33
description: A Helm chart for Kubernetes
44
type: application
5-
version: 0.1.1
5+
version: 0.1.2
66
appVersion: "v0.1.0-rc.3"

‎helm/netbird-operator/templates/_helpers.tpl‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ caCert: {{ index $secret.data "ca.crt" }}
9292
clientCert: {{ index $secret.data "tls.crt" }}
9393
clientKey: {{ index $secret.data "tls.key" }}
9494
{{- else -}}
95-
{{- $altNames := list (printf "%s.%s" $serviceName .Release.Namespace) (printf "%s.%s.svc" $serviceName .Release.Namespace) (printf "%s.%s.svc.%s" $serviceName .Release.Namespace .Values.webhook.cluster.dnsDomain) -}}
95+
{{- $altNames := list (printf "%s.%s" $serviceName .Release.Namespace) (printf "%s.%s.svc" $serviceName .Release.Namespace) (printf "%s.%s.%s" $serviceName .Release.Namespace .Values.cluster.dns) -}}
9696
{{- $ca := genCA "netbird-operator-ca" 3650 -}}
9797
{{- $cert := genSignedCert (include "netbird-operator.fullname" .) nil $altNames 3650 $ca -}}
9898
caCert: {{ $ca.Cert | b64enc }}

‎helm/netbird-operator/templates/webhook.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -235,7 +235,7 @@ metadata:
235235
spec:
236236
dnsNames:
237237
- {{ template "netbird-operator.webhookService" . }}.{{ .Release.Namespace }}.svc
238-
- {{ template "netbird-operator.webhookService" . }}.{{ .Release.Namespace }}.svc.{{ .Values.webhook.cluster.dnsDomain }}
238+
- {{ template "netbird-operator.webhookService" . }}.{{ .Release.Namespace }}.{{ .Values.cluster.dns }}
239239
issuerRef:
240240
kind: Issuer
241241
name: {{ template "netbird-operator.fullname" . }}-selfsigned-issuer

‎helm/netbird-operator/values.yaml‎

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
clusterSecretsPermissions:
2+
# Required for NBSetupKey validation
3+
# Required for Ingress functionality to create and validate secrets for routing peers
24
allowAllSecrets: true
35

46
webhook:
@@ -7,22 +9,21 @@ webhook:
79
port: 443
810
targetPort: 9443
911

10-
cluster:
11-
# Cluster DNS domain (required for requesting TLS certificates)
12-
dnsDomain: cluster.local
13-
1412
# TLS configuration for webhook
13+
# Optional, unused if webhook.enableCertManager is set to true
1514
tls: {}
1615

17-
# Use cert-manager to provision webhook certificates
16+
# Use cert-manager to provision webhook certificates (recommended)
1817
enableCertManager: true
1918

19+
# Narrow down validation and mutation webhooks namespaces
2020
namespaceSelectors: []
2121
# - key: foo
2222
# operator: In
2323
# values:
2424
# - bar
2525

26+
# Narrow down validation and mutation webhooks objects
2627
objectSelector:
2728
matchExpressions: []
2829
# - key: app.kubernetes.io/name
@@ -130,9 +131,12 @@ operator:
130131
affinity: {}
131132

132133
ingress:
134+
# Enable ingress capabilities to expose services
133135
enabled: false
136+
# Create router per namespace, useful for strict networking requirements
134137
namespacedNetworks: false
135138
router:
139+
# Deploy routing peer(s)
136140
enabled: false
137141
# replicas: 3
138142
# resources:
@@ -146,8 +150,8 @@ ingress:
146150
# annotations: {}
147151
# nodeSelector: {}
148152
# tolerations: []
149-
# Only needed for namespacedNetworks
150-
# namespaces:
153+
# Only needed if namespacedNetworks is set to true
154+
namespaces: {}
151155
# default:
152156
# replicas: 3
153157
# resources:
@@ -161,16 +165,20 @@ ingress:
161165
# annotations: {}
162166
# nodeSelector: {}
163167
# tolerations: []
168+
# NetBird Policies for use with exposed services
164169
policies: {}
165170
# default:
166171
# name: Kubernetes Default Policy
167172
# sourceGroups:
168173
# - All
169174

170-
cluster: {}
175+
cluster:
176+
# Cluster DNS name (used for webhooks certificates and for network resource DNS names)
177+
dns: svc.cluster.local
178+
# Cluster name (used for generating network and network resource names in NetBird)
171179
# name: kubernetes
172-
# dns: svc.cluster.local
173180

174181
netbirdAPI: {}
182+
# NetBird Service Account Token
175183
# key: "nbp_m0LM9ZZvDUzFO0pY50iChDOTxJgKFM3DIqmZ"
176184
# keyFromSecret: "Secret name with NB_API_KEY=Service Account Token"

‎internal/controller/nbgroup_controller.go‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,11 @@ type NBGroupReconciler struct {
2828
}
2929

3030
const (
31+
// defaultRequeueAfter default requeue duration
32+
// due to controller-runtime limitations, sync periods may reach up to 10 hours if no changes are detected
33+
// in watched resources.
34+
// This may cause issues when NetBird-side resources are out-of-sync and need to be reconciled, this is a temporary
35+
// fix to this issue by syncing with NetBird more frequently.
3136
defaultRequeueAfter = 15 * time.Minute
3237
)
3338

@@ -69,7 +74,9 @@ func (r *NBGroupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (re
6974
return r.syncNetBirdGroup(ctx, &nbGroup, logger)
7075
}
7176

77+
// syncNetBirdGroup reconcilation logic for non-deleted objects.
7278
func (r *NBGroupReconciler) syncNetBirdGroup(ctx context.Context, nbGroup *netbirdiov1.NBGroup, logger logr.Logger) (ctrl.Result, error) {
79+
// Get all NetBird groups to ensure no group duplication
7380
groups, err := r.netbird.Groups.List(ctx)
7481
if err != nil {
7582
logger.Error(errNetBirdAPI, "error listing groups", "err", err)
@@ -81,6 +88,8 @@ func (r *NBGroupReconciler) syncNetBirdGroup(ctx context.Context, nbGroup *netbi
8188
group = &g
8289
}
8390
}
91+
92+
// Create group if not exists, and update status.groupId
8493
if nbGroup.Status.GroupID == nil && group == nil {
8594
logger.Info("NBGroup: Creating group on NetBird", "name", nbGroup.Spec.Name)
8695
group, err := r.netbird.Groups.Create(ctx, api.GroupRequest{
@@ -118,6 +127,7 @@ func (r *NBGroupReconciler) syncNetBirdGroup(ctx context.Context, nbGroup *netbi
118127
}
119128

120129
func (r *NBGroupReconciler) handleDelete(ctx context.Context, nbGroup netbirdiov1.NBGroup, logger logr.Logger) error {
130+
// Group doesn't exist on NetBird, no need for cleanup
121131
if nbGroup.Status.GroupID == nil {
122132
nbGroup.Finalizers = util.Without(nbGroup.Finalizers, "netbird.io/group-cleanup")
123133
err := r.Client.Update(ctx, &nbGroup)
@@ -160,6 +170,9 @@ func (r *NBGroupReconciler) handleDelete(ctx context.Context, nbGroup netbirdiov
160170
return nil
161171
}
162172
}
173+
174+
// No other NBGroup with same name on the cluster
175+
// This could be a group created by user elsewhere or some resources belonging to the group are still deleting.
163176
return err
164177
}
165178

0 commit comments

Comments
 (0)