From 4ad239b35a23caae15f10171b962fe92bc4d6475 Mon Sep 17 00:00:00 2001 From: M Essam Hamed Date: Sun, 2 Mar 2025 13:13:56 +0200 Subject: [PATCH 1/4] Add documentation --- README.md | 92 ++++---------- docs/usage.md | 112 ++++++++++++++++++ examples/ingress/exposed-nginx.yaml | 47 ++++++++ examples/ingress/values.yaml | 12 ++ examples/setup-keys/example.yaml | 49 ++++++++ helm/kubernetes-operator/Chart.yaml | 2 +- .../templates/_helpers.tpl | 2 +- .../templates/webhook.yaml | 2 +- helm/kubernetes-operator/values.yaml | 26 ++-- internal/controller/nbgroup_controller.go | 13 ++ internal/controller/nbpolicy_controller.go | 11 +- internal/controller/nbresource_controller.go | 13 ++ .../controller/nbroutingpeer_controller.go | 13 +- internal/controller/service_controller.go | 4 + 14 files changed, 311 insertions(+), 87 deletions(-) create mode 100644 docs/usage.md create mode 100644 examples/ingress/exposed-nginx.yaml create mode 100644 examples/ingress/values.yaml create mode 100644 examples/setup-keys/example.yaml diff --git a/README.md b/README.md index 3c46956..1c34577 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # NetBird Kubernetes Operator For easily provisioning access to Kubernetes resources using NetBird. +https://github.com/user-attachments/assets/5472a499-e63d-4301-a513-ad84cfe5ca7b + ## Description This operator enables easily provisioning NetBird access on kubernetes clusters, allowing users to access internal resources directly. @@ -8,94 +10,40 @@ This operator enables easily provisioning NetBird access on kubernetes clusters, ## Getting Started ### Prerequisites -- helm version 3+ +- (Recommended) helm version 3+ - kubectl version v1.11.3+. - Access to a Kubernetes v1.11.3+ cluster. -- (Optional for Helm chart installation) Cert Manager. - -### To Deploy on the cluster +- (Recommended) Cert Manager. -**Using the install.yaml** -```sh -kubectl create namespace netbird -kubectl apply -n netbird -f https://github.com/netbirdio/kubernetes-operator/releases/latest/manifests/install.yaml -``` +### Deployment +> [!NOTE] +> Helm Installation method is recommended due to automation of multiple settings within the deployment. -**Using the Helm Chart** +#### Using Helm +1. Add helm repository. ```sh helm repo add netbirdio https://netbirdio.github.io/kubernetes-operator -helm install -n netbird kubernetes-operator netbirdio/kubernetes-operator ``` +2. (Recommended) Install [cert-manager](https://cert-manager.io/docs/installation/#default-static-install). +1. (Recommended) Create a values.yaml file, check `helm show values netbirdio/kubernetes-operator` for more info. +1. Install using `helm install --create-namespace -f values.yaml -n netbird netbird-operator netbirdio/kubernetes-operator`. -For more options, check the default values by running -```sh -helm show values netbirdio/kubernetes-operator -``` +#### Using install.yaml -### To Uninstall -**Using install.yaml** +> [!IMPORTANT] +> install.yaml only includes a very basic template for deploying a stripped down version of kubernetes-operator. +> This excludes any and all configuration for ingress capabilities, and requires cert-manager to be installed. ```sh -kubectl delete -n netbird -f https://github.com/netbirdio/kubernetes-operator/releases/latest/manifests/install.yaml -kubectl delete namespace netbird -``` - -**Using helm** - -```sh -helm uninstall -n netbird kubernetes-operator +kubectl create namespace netbird +kubectl apply -n netbird -f https://raw.githubusercontent.com/netbirdio/kubernetes-operator/refs/heads/main/manifests/install.yaml ``` -### Provision pods with NetBird access - -1. Create a Setup Key in your [NetBird console](https://docs.netbird.io/how-to/register-machines-using-setup-keys#using-setup-keys). -1. Create a Secret object in the namespace where you need to provision NetBird access (secret name and field can be anything). -```yaml -apiVersion: v1 -stringData: - setupkey: EEEEEEEE-EEEE-EEEE-EEEE-EEEEEEEEEEEE -kind: Secret -metadata: - name: test -``` -1. Create an NBSetupKey object referring to your secret. -```yaml -apiVersion: netbird.io/v1 -kind: NBSetupKey -metadata: - name: test -spec: - # Optional, overrides management URL for this setupkey only - # defaults to https://api.netbird.io - managementURL: https://netbird.example.com - secretKeyRef: - name: test # Required - key: setupkey # Required -``` -1. Annotate the pods you need to inject NetBird into with `netbird.io/setup-key`. -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: deployment -spec: - selector: - matchLabels: - app: myapp - template: - metadata: - labels: - app: myapp - annotations: - netbird.io/setup-key: test # Must match the name of an NBSetupKey object in the same namespace - spec: - containers: - - image: yourimage - name: container +### Usage -``` +Checks [usage.md](docs/usage.md). ## Contributing diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..8e68da6 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,112 @@ +# Usage + +## Provision pods with NetBird access + +1. Create a Setup Key in your [NetBird console](https://docs.netbird.io/how-to/register-machines-using-setup-keys#using-setup-keys). +1. Create a Secret object in the namespace where you need to provision NetBird access (secret name and field can be anything). +```yaml +apiVersion: v1 +stringData: + setupkey: EEEEEEEE-EEEE-EEEE-EEEE-EEEEEEEEEEEE +kind: Secret +metadata: + name: test +``` +1. Create an NBSetupKey object referring to your secret. +```yaml +apiVersion: netbird.io/v1 +kind: NBSetupKey +metadata: + name: test +spec: + # Optional, overrides management URL for this setupkey only + # defaults to https://api.netbird.io + managementURL: https://netbird.example.com + secretKeyRef: + name: test # Required + key: setupkey # Required +``` +1. Annotate the pods you need to inject NetBird into with `netbird.io/setup-key`. +```yaml +kind: Deployment +... +spec: +... + template: + metadata: + annotations: + netbird.io/setup-key: test # Must match the name of an NBSetupKey object in the same namespace +... + spec: + containers: +... +``` + +## Provisioning Networks (Ingress Functionality) + +### Granting controller access to NetBird Management + +> [!IMPORTANT] +> NetBird kubernetes operator generates configurations using NetBird API, editing or deleting these configurations in the NetBird console may cause temporary network disconnection until the operator reconciles the configuration. + +1. Create a Service User on your NetBird dashboard (Must be Admin). [Doc](https://docs.netbird.io/how-to/access-netbird-public-api#creating-a-service-user). +1. Create access token for the Service User (Must be Admin). [Doc](https://docs.netbird.io/how-to/access-netbird-public-api#creating-a-service-user). +1. Add access token to your helm values file under `netbirdAPI.key`. + 1. Alternatively, provision secret in the same namespace as the operator and set the key `NB_API_KEY` to the access token generated. + 1. Set `netbirdAPI.keyFromSecret` to the name of the secret created. +1. Set `ingress.enabled` to `true`. + 1. Optionally, to provision network immediately, set `ingress.router.enabled` to `true`. + 1. Optionally, to provision 1 network per kubernetes namespace, set `ingress.namespacedNetworks` to `true`. +1. Run `helm install` or `helm upgrade`. + +### Exposing a Service + +> [!IMPORTANT] +> Ingress DNS Resolution requires DNS Wildcard Routing to be enabled, and at least one DNS Nameserver configured for clients. + +|Annotation|Description|Default|Valid Values| +|---|---|---|---| +|`netbird.io/expose`|Expose service using NetBird Network Resource||(`null`, `true`)| +|`netbird.io/groups`|Comma-separated list of group names to assign to Network Resource|`{ClusterName}-{Namespace}-{Service}`|Any comma-separated list of strings.| +|`netbird.io/resource-name`|Network Resource name|`{Namespace}-{Service}`|Any valid network resource name, make sure they're unique!| +|`netbird.io/policy`|Name of NBPolicy to propagate service ports as destination.||Name of any NBPolicy resource| +|`netbird.io/policy-ports`|Narrow down exposed ports in policy, leave empty for all ports.||Comma-separated integer list, integers must be between 0-65535| +|`netbird.io/policy-protocol`|Narrow down protocol for use in policy, leave empty for all protocols.||(`tcp`,`udp`)| + +### Notes +* `netbird.io/expose` will interpret any string as true value, the only false value is `null`. +* The operator does **not** handle duplicate resource names within the same network, it is up to you to ensure resource names are unique within the same network. +* While the NetBird console will allow group names to contain commas, this is not allowed in `netbird.io/groups` annotation as commas are used as separators. +* If a group already exists on NetBird console with the same name, NetBird Operator will use that group ID instead of creating a new group. +* NetBird Operator will attempt to clean up any resources created, including groups created for resources. + * In case a group is used by resources that cannot be cleaned up by the operator, the operator will eventually ignore the group in NetBird. + * It's recommended to use unique groups per NetBird Operator installation to remove any possible conflicts. +* NetBird Operator does not validate service annotations on update, as this may cause unnecessary overhead on any Service update. + +### Managing Policies + +Simply add policies under `ingress.policies`, for example: +1. Add the following configuration in your `values.yaml` file. +```yaml +ingress: + policies: + default: + name: Kubernetes Default Policy # Required, name of policy in NetBird console + description: Default # Optional + sourceGroups: # Required, name of groups to assign as source in Policy. + - All + ports: # Optional, resources annotated 'netbird.io/policy=default' will append to this. + - 443 + protocols: # Optional, restricts protocols allowed to resources, defaults to ['tcp', 'udp']. + - tcp + - udp + bidirectional: true # Optional, defaults to true +``` +2. Reference policy in Services using `netbird.io/policy=default`, this will add relevant ports and destination groups to Policy. +3. (Optional) limit specific ports in exposed service by adding `netbird.io/policy-ports=443`. +4. (Optional) limit specific protocol in exposed service by adding `netbird.io/policy-protocol=tcp`. + +#### Notes +* Each NBPolicy will only create policies in NetBird console when information provided is enough to create one. If there are no services acting as destination, or specified services do not conform to the protocol(s) defined, policy will not be created. +* Each NBPolicy will create one Policy in NetBird console per protocol specified as long as protocol has destinations, this ensures better secured policies by separating ports for TCP and UDP. +* Policies currently do not support ICMP protocol, as ICMP is not supported in kubernetes services, and there are [no current plans to support it](https://discuss.kubernetes.io/t/icmp-support-for-kubernetes-service/21738). diff --git a/examples/ingress/exposed-nginx.yaml b/examples/ingress/exposed-nginx.yaml new file mode 100644 index 0000000..6e44d2a --- /dev/null +++ b/examples/ingress/exposed-nginx.yaml @@ -0,0 +1,47 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app: test + name: test + namespace: default +spec: + replicas: 1 + selector: + matchLabels: + app: test + strategy: + rollingUpdate: + maxSurge: 25% + maxUnavailable: 25% + type: RollingUpdate + template: + metadata: + labels: + app: test + spec: + containers: + - image: nginx + imagePullPolicy: Always + name: nginx +--- +apiVersion: v1 +kind: Service +metadata: + annotations: + netbird.io/expose: "true" + netbird.io/policy: default + netbird.io/resource-name: nginx + labels: + app: test + name: test + namespace: default +spec: + ports: + - name: http + port: 80 + protocol: TCP + targetPort: 80 + selector: + app: test + type: ClusterIP diff --git a/examples/ingress/values.yaml b/examples/ingress/values.yaml new file mode 100644 index 0000000..1139206 --- /dev/null +++ b/examples/ingress/values.yaml @@ -0,0 +1,12 @@ +ingress: + enabled: true + router: + enabled: true + policies: + default: + name: Kubernetes Default Policy + sourceGroups: + - All + +netbirdAPI: + key: "nbp_m0LM9ZZvDUzFO0pY50iChDOTxJgKFM3DIqmZ" # Replace with valid NetBird Service Account token diff --git a/examples/setup-keys/example.yaml b/examples/setup-keys/example.yaml new file mode 100644 index 0000000..a2a8a8a --- /dev/null +++ b/examples/setup-keys/example.yaml @@ -0,0 +1,49 @@ +apiVersion: v1 +kind: Secret +metadata: + name: test + namespace: default +stringData: + SETUP_KEY: 50445ABC-8901-4050-8047-0A390658A79B # Replace with valid setup key +--- +apiVersion: netbird.io/v1 +kind: NBSetupKey +metadata: + name: test + namespace: default +spec: + secretKeyRef: + name: test + key: SETUP_KEY +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app: test + name: test + namespace: default +spec: + replicas: 1 + selector: + matchLabels: + app: test + strategy: + rollingUpdate: + maxSurge: 25% + maxUnavailable: 25% + type: RollingUpdate + template: + metadata: + labels: + app: test + annotations: + netbird.io/setup-key: test + spec: + containers: + - image: ubuntu + imagePullPolicy: Always + name: ubuntu + command: + - sleep + - inf diff --git a/helm/kubernetes-operator/Chart.yaml b/helm/kubernetes-operator/Chart.yaml index 10e2eb3..370c351 100644 --- a/helm/kubernetes-operator/Chart.yaml +++ b/helm/kubernetes-operator/Chart.yaml @@ -2,5 +2,5 @@ apiVersion: v2 name: kubernetes-operator description: A Helm chart for Kubernetes type: application -version: 0.1.1 +version: 0.1.2 appVersion: "v0.1.0-rc.3" diff --git a/helm/kubernetes-operator/templates/_helpers.tpl b/helm/kubernetes-operator/templates/_helpers.tpl index e80a9cf..2c98e59 100644 --- a/helm/kubernetes-operator/templates/_helpers.tpl +++ b/helm/kubernetes-operator/templates/_helpers.tpl @@ -92,7 +92,7 @@ caCert: {{ index $secret.data "ca.crt" }} clientCert: {{ index $secret.data "tls.crt" }} clientKey: {{ index $secret.data "tls.key" }} {{- else -}} -{{- $altNames := list (printf "%s.%s" $serviceName .Release.Namespace) (printf "%s.%s.svc" $serviceName .Release.Namespace) (printf "%s.%s.svc.%s" $serviceName .Release.Namespace .Values.webhook.cluster.dnsDomain) -}} +{{- $altNames := list (printf "%s.%s" $serviceName .Release.Namespace) (printf "%s.%s.svc" $serviceName .Release.Namespace) (printf "%s.%s.%s" $serviceName .Release.Namespace .Values.cluster.dns) -}} {{- $ca := genCA "kubernetes-operator-ca" 3650 -}} {{- $cert := genSignedCert (include "kubernetes-operator.fullname" .) nil $altNames 3650 $ca -}} caCert: {{ $ca.Cert | b64enc }} diff --git a/helm/kubernetes-operator/templates/webhook.yaml b/helm/kubernetes-operator/templates/webhook.yaml index 5ac8055..4ba96f3 100644 --- a/helm/kubernetes-operator/templates/webhook.yaml +++ b/helm/kubernetes-operator/templates/webhook.yaml @@ -235,7 +235,7 @@ metadata: spec: dnsNames: - {{ template "kubernetes-operator.webhookService" . }}.{{ .Release.Namespace }}.svc - - {{ template "kubernetes-operator.webhookService" . }}.{{ .Release.Namespace }}.svc.{{ .Values.webhook.cluster.dnsDomain }} + - {{ template "kubernetes-operator.webhookService" . }}.{{ .Release.Namespace }}.{{ .Values.cluster.dns }} issuerRef: kind: Issuer name: {{ template "kubernetes-operator.fullname" . }}-selfsigned-issuer diff --git a/helm/kubernetes-operator/values.yaml b/helm/kubernetes-operator/values.yaml index 24f2167..d512b2d 100644 --- a/helm/kubernetes-operator/values.yaml +++ b/helm/kubernetes-operator/values.yaml @@ -1,4 +1,6 @@ clusterSecretsPermissions: + # Required for NBSetupKey validation + # Required for Ingress functionality to create and validate secrets for routing peers allowAllSecrets: true webhook: @@ -7,22 +9,21 @@ webhook: port: 443 targetPort: 9443 - cluster: - # Cluster DNS domain (required for requesting TLS certificates) - dnsDomain: cluster.local - # TLS configuration for webhook + # Optional, unused if webhook.enableCertManager is set to true tls: {} - # Use cert-manager to provision webhook certificates + # Use cert-manager to provision webhook certificates (recommended) enableCertManager: true + # Narrow down validation and mutation webhooks namespaces namespaceSelectors: [] # - key: foo # operator: In # values: # - bar + # Narrow down validation and mutation webhooks objects objectSelector: matchExpressions: [] # - key: app.kubernetes.io/name @@ -130,9 +131,12 @@ operator: affinity: {} ingress: + # Enable ingress capabilities to expose services enabled: false + # Create router per namespace, useful for strict networking requirements namespacedNetworks: false router: + # Deploy routing peer(s) enabled: false # replicas: 3 # resources: @@ -146,8 +150,8 @@ ingress: # annotations: {} # nodeSelector: {} # tolerations: [] - # Only needed for namespacedNetworks - # namespaces: + # Only needed if namespacedNetworks is set to true + namespaces: {} # default: # replicas: 3 # resources: @@ -161,16 +165,20 @@ ingress: # annotations: {} # nodeSelector: {} # tolerations: [] + # NetBird Policies for use with exposed services policies: {} # default: # name: Kubernetes Default Policy # sourceGroups: # - All -cluster: {} +cluster: + # Cluster DNS name (used for webhooks certificates and for network resource DNS names) + dns: svc.cluster.local + # Cluster name (used for generating network and network resource names in NetBird) # name: kubernetes - # dns: svc.cluster.local netbirdAPI: {} + # NetBird Service Account Token # key: "nbp_m0LM9ZZvDUzFO0pY50iChDOTxJgKFM3DIqmZ" # keyFromSecret: "Secret name with NB_API_KEY=Service Account Token" \ No newline at end of file diff --git a/internal/controller/nbgroup_controller.go b/internal/controller/nbgroup_controller.go index 1de3bc0..a7d2b3a 100644 --- a/internal/controller/nbgroup_controller.go +++ b/internal/controller/nbgroup_controller.go @@ -28,6 +28,11 @@ type NBGroupReconciler struct { } const ( + // defaultRequeueAfter default requeue duration + // due to controller-runtime limitations, sync periods may reach up to 10 hours if no changes are detected + // in watched resources. + // This may cause issues when NetBird-side resources are out-of-sync and need to be reconciled, this is a temporary + // fix to this issue by syncing with NetBird more frequently. defaultRequeueAfter = 15 * time.Minute ) @@ -69,7 +74,9 @@ func (r *NBGroupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (re return r.syncNetBirdGroup(ctx, &nbGroup, logger) } +// syncNetBirdGroup reconciliation logic for non-deleted objects. func (r *NBGroupReconciler) syncNetBirdGroup(ctx context.Context, nbGroup *netbirdiov1.NBGroup, logger logr.Logger) (ctrl.Result, error) { + // Get all NetBird groups to ensure no group duplication groups, err := r.netbird.Groups.List(ctx) if err != nil { logger.Error(errNetBirdAPI, "error listing groups", "err", err) @@ -81,6 +88,8 @@ func (r *NBGroupReconciler) syncNetBirdGroup(ctx context.Context, nbGroup *netbi group = &g } } + + // Create group if not exists, and update status.groupId if nbGroup.Status.GroupID == nil && group == nil { logger.Info("NBGroup: Creating group on NetBird", "name", nbGroup.Spec.Name) group, err := r.netbird.Groups.Create(ctx, api.GroupRequest{ @@ -118,6 +127,7 @@ func (r *NBGroupReconciler) syncNetBirdGroup(ctx context.Context, nbGroup *netbi } func (r *NBGroupReconciler) handleDelete(ctx context.Context, nbGroup netbirdiov1.NBGroup, logger logr.Logger) error { + // Group doesn't exist on NetBird, no need for cleanup if nbGroup.Status.GroupID == nil { nbGroup.Finalizers = util.Without(nbGroup.Finalizers, "netbird.io/group-cleanup") err := r.Client.Update(ctx, &nbGroup) @@ -160,6 +170,9 @@ func (r *NBGroupReconciler) handleDelete(ctx context.Context, nbGroup netbirdiov return nil } } + + // No other NBGroup with same name on the cluster + // This could be a group created by user elsewhere or some resources belonging to the group are still deleting. return err } diff --git a/internal/controller/nbpolicy_controller.go b/internal/controller/nbpolicy_controller.go index ed49475..4621d05 100644 --- a/internal/controller/nbpolicy_controller.go +++ b/internal/controller/nbpolicy_controller.go @@ -35,6 +35,7 @@ var ( errNetBirdAPI = fmt.Errorf("netbird API error") ) +// getResources get all NBResource objects in policy.status.managedServiceList func (r *NBPolicyReconciler) getResources(ctx context.Context, nbPolicy *netbirdiov1.NBPolicy, logger logr.Logger) ([]netbirdiov1.NBResource, error) { var resourceList []netbirdiov1.NBResource var updatedManagedServiceList []string @@ -58,6 +59,8 @@ func (r *NBPolicyReconciler) getResources(ctx context.Context, nbPolicy *netbird return resourceList, nil } +// mapResources map each NBResource ports and protocols into one object to generate the policy +// returns map[protocol] => ports, destination group IDs func (r *NBPolicyReconciler) mapResources(ctx context.Context, nbPolicy *netbirdiov1.NBPolicy, resources []netbirdiov1.NBResource, logger logr.Logger) (map[string][]int32, []string, error) { portMapping := map[string]map[int32]interface{}{ "tcp": make(map[int32]interface{}), @@ -92,6 +95,7 @@ func (r *NBPolicyReconciler) mapResources(ctx context.Context, nbPolicy *netbird return ports, groups, nil } +// createPolicy helper for creating policy with settings func (r *NBPolicyReconciler) createPolicy(ctx context.Context, nbPolicy *netbirdiov1.NBPolicy, protocol string, sourceGroupIDs, destinationGroupIDs, ports []string, logger logr.Logger) (*string, error) { policyName := fmt.Sprintf("%s %s", nbPolicy.Spec.Name, strings.ToUpper(protocol)) logger.Info("Creating NetBird Policy", "name", policyName, "description", nbPolicy.Spec.Description, "protocol", protocol, "sources", sourceGroupIDs, "destinations", destinationGroupIDs, "ports", ports, "bidirectional", nbPolicy.Spec.Bidirectional) @@ -123,6 +127,7 @@ func (r *NBPolicyReconciler) createPolicy(ctx context.Context, nbPolicy *netbird return policy.Id, nil } +// updatePolicy helper for updating policy with settings func (r *NBPolicyReconciler) updatePolicy(ctx context.Context, policyID *string, nbPolicy *netbirdiov1.NBPolicy, protocol string, sourceGroupIDs, destinationGroupIDs, ports []string, logger logr.Logger) (*string, bool, error) { policyName := fmt.Sprintf("%s %s", nbPolicy.Spec.Name, strings.ToUpper(protocol)) logger.Info("Updating NetBird Policy", "name", policyName, "description", nbPolicy.Spec.Description, "protocol", protocol, "sources", sourceGroupIDs, "destinations", destinationGroupIDs, "ports", ports, "bidirectional", nbPolicy.Spec.Bidirectional) @@ -221,7 +226,7 @@ func (r *NBPolicyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (r return ctrl.Result{}, err } - requeue, err := r.handlePolicies(ctx, &nbPolicy, sourceGroupIDs, destGroups, portMapping, logger) + requeue, err := r.syncPolicy(ctx, &nbPolicy, sourceGroupIDs, destGroups, portMapping, logger) if requeue || err != nil { return ctrl.Result{Requeue: requeue}, err @@ -232,7 +237,8 @@ func (r *NBPolicyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (r return ctrl.Result{}, nil } -func (r *NBPolicyReconciler) handlePolicies(ctx context.Context, nbPolicy *netbirdiov1.NBPolicy, sourceGroups, destGroups []string, portMapping map[string][]int32, logger logr.Logger) (bool, error) { +// syncPolicy ensure upstream policy is up-to-date +func (r *NBPolicyReconciler) syncPolicy(ctx context.Context, nbPolicy *netbirdiov1.NBPolicy, sourceGroups, destGroups []string, portMapping map[string][]int32, logger logr.Logger) (bool, error) { requeue := false for protocol, ports := range portMapping { @@ -340,6 +346,7 @@ func (r *NBPolicyReconciler) handleDelete(ctx context.Context, nbPolicy netbirdi return nil } +// groupNamesToIDs map list of NetBird group names to group IDs func (r *NBPolicyReconciler) groupNamesToIDs(ctx context.Context, groupNames []string, logger logr.Logger) ([]string, error) { groups, err := r.netbird.Groups.List(ctx) if err != nil { diff --git a/internal/controller/nbresource_controller.go b/internal/controller/nbresource_controller.go index cf15e3b..666febe 100644 --- a/internal/controller/nbresource_controller.go +++ b/internal/controller/nbresource_controller.go @@ -99,6 +99,7 @@ func (r *NBResourceReconciler) Reconcile(ctx context.Context, req ctrl.Request) return ctrl.Result{}, nil } +// handlePolicy update NBPolicy if defined to add self reference to policy status func (r *NBResourceReconciler) handlePolicy(ctx context.Context, req ctrl.Request, nbResource *netbirdiov1.NBResource, groupIDs []string, logger logr.Logger) error { if nbResource.Status.PolicyName == nil && nbResource.Spec.PolicyName == "" { return nil @@ -108,6 +109,7 @@ func (r *NBResourceReconciler) handlePolicy(ctx context.Context, req ctrl.Reques var nbPolicy netbirdiov1.NBPolicy if nbResource.Spec.PolicyName == "" && nbResource.Status.PolicyName != nil { + // Remove self reference from policy status nbResource.Status.PolicyName = nil err := r.Client.Get(ctx, types.NamespacedName{Name: *nbResource.Status.PolicyName}, &nbPolicy) if err != nil { @@ -120,6 +122,8 @@ func (r *NBResourceReconciler) handlePolicy(ctx context.Context, req ctrl.Reques updatePolicyStatus = true } } else { + // Update policy settings if any difference is found + // TODO: Handle updated policy name by removing reference from old policy name in status.policyName err := r.Client.Get(ctx, types.NamespacedName{Name: nbResource.Spec.PolicyName}, &nbPolicy) if err != nil { logger.Error(errKubernetesAPI, "error getting NBPolicy", "err", err, "policyName", nbResource.Spec.PolicyName) @@ -166,6 +170,7 @@ func (r *NBResourceReconciler) handlePolicy(ctx context.Context, req ctrl.Reques return nil } +// handleGroupUpdate update network resource groups func (r *NBResourceReconciler) handleGroupUpdate(ctx context.Context, nbResource *netbirdiov1.NBResource, groupIDs []string, resource *api.NetworkResource, logger logr.Logger) error { // Handle possible updated group IDs groupIDMap := make(map[string]interface{}) @@ -198,6 +203,7 @@ func (r *NBResourceReconciler) handleGroupUpdate(ctx context.Context, nbResource return nil } +// handleNetBirdResource sync NetBird Network Resource func (r *NBResourceReconciler) handleNetBirdResource(ctx context.Context, nbResource *netbirdiov1.NBResource, groupIDs []string, logger logr.Logger) (*api.NetworkResource, error) { var resource *api.NetworkResource var err error @@ -208,6 +214,8 @@ func (r *NBResourceReconciler) handleNetBirdResource(ctx context.Context, nbReso return nil, err } } + + // Create/Update upstream network resource if nbResource.Status.NetworkResourceID == nil && resource == nil { resource, err := r.netbird.Networks.Resources(nbResource.Spec.NetworkID).Create(ctx, api.NetworkResourceRequest{ Address: nbResource.Spec.Address, @@ -255,6 +263,7 @@ func (r *NBResourceReconciler) handleNetBirdResource(ctx context.Context, nbReso return resource, nil } +// handleGroups create NBGroup objects for each group specified in NBResource func (r *NBResourceReconciler) handleGroups(ctx context.Context, req ctrl.Request, nbResource *netbirdiov1.NBResource, logger logr.Logger) ([]string, *ctrl.Result, error) { var groupIDs []string @@ -267,6 +276,7 @@ func (r *NBResourceReconciler) handleGroups(ctx context.Context, req ctrl.Reques logger.Error(errKubernetesAPI, "error getting NBGroup", "err", err) return nil, &ctrl.Result{}, err } else if errors.IsNotFound(err) { + // Create NBGroup nbGroup = netbirdiov1.NBGroup{ ObjectMeta: v1.ObjectMeta{ Name: groupNameRFC, @@ -295,6 +305,7 @@ func (r *NBResourceReconciler) handleGroups(ctx context.Context, req ctrl.Reques continue } else { + // Add NBResource as owner to NBGroup if not already done ownerExists := false for _, o := range nbGroup.OwnerReferences { if o.UID == nbResource.UID { @@ -324,6 +335,7 @@ func (r *NBResourceReconciler) handleGroups(ctx context.Context, req ctrl.Reques } } + // if not all groups are ready, requeue if len(groupIDs) != len(nbResource.Spec.Groups) { return nil, &ctrl.Result{RequeueAfter: 5 * time.Second}, nil } @@ -368,6 +380,7 @@ func (r *NBResourceReconciler) handleDelete(ctx context.Context, req ctrl.Reques } for _, g := range nbGroupList.Items { + // TODO: Handle multiple owners if len(g.OwnerReferences) > 0 && g.OwnerReferences[0].UID == nbResource.UID { g.Finalizers = util.Without(g.Finalizers, "netbird.io/resource-cleanup") err = r.Client.Update(ctx, &g) diff --git a/internal/controller/nbroutingpeer_controller.go b/internal/controller/nbroutingpeer_controller.go index c4a3923..eaac7ae 100644 --- a/internal/controller/nbroutingpeer_controller.go +++ b/internal/controller/nbroutingpeer_controller.go @@ -106,6 +106,7 @@ func (r *NBRoutingPeerReconciler) Reconcile(ctx context.Context, req ctrl.Reques return ctrl.Result{}, nil } +// handleDeployment reconcile routing peer Deployment func (r *NBRoutingPeerReconciler) handleDeployment(ctx context.Context, req ctrl.Request, nbrp *netbirdiov1.NBRoutingPeer, logger logr.Logger) error { routingPeerDeployment := appsv1.Deployment{} err := r.Client.Get(ctx, req.NamespacedName, &routingPeerDeployment) @@ -115,6 +116,7 @@ func (r *NBRoutingPeerReconciler) handleDeployment(ctx context.Context, req ctrl return err } + // Create deployment if errors.IsNotFound(err) { var replicas int32 = 3 if nbrp.Spec.Replicas != nil { @@ -253,6 +255,7 @@ func (r *NBRoutingPeerReconciler) handleDeployment(ctx context.Context, req ctrl patch := client.StrategicMergeFrom(&routingPeerDeployment) bs, _ := patch.Data(updatedDeployment) + // To ensure no useless patching is done to the deployment being watched // Minimum patch size is 2 for "{}" if len(bs) <= 2 { return nil @@ -268,6 +271,7 @@ func (r *NBRoutingPeerReconciler) handleDeployment(ctx context.Context, req ctrl return nil } +// handleRouter reconcile network routing peer in NetBird management API func (r *NBRoutingPeerReconciler) handleRouter(ctx context.Context, nbrp *netbirdiov1.NBRoutingPeer, nbGroup netbirdiov1.NBGroup, logger logr.Logger) error { // Check NetworkRouter exists routers, err := r.netbird.Networks.Routers(*nbrp.Status.NetworkID).List(ctx) @@ -280,8 +284,10 @@ func (r *NBRoutingPeerReconciler) handleRouter(ctx context.Context, nbrp *netbir if nbrp.Status.RouterID == nil || len(routers) == 0 { if len(routers) > 0 { + // Router exists but isn't saved to status nbrp.Status.RouterID = &routers[0].Id } else { + // Create network router router, err := r.netbird.Networks.Routers(*nbrp.Status.NetworkID).Create(ctx, api.NetworkRouterRequest{ Enabled: true, Masquerade: true, @@ -298,6 +304,7 @@ func (r *NBRoutingPeerReconciler) handleRouter(ctx context.Context, nbrp *netbir nbrp.Status.RouterID = &router.Id } } else { + // Ensure network router settings are correct if !routers[0].Enabled || !routers[0].Masquerade || routers[0].Metric != 9999 || len(*routers[0].PeerGroups) != 1 || (*routers[0].PeerGroups)[0] != *nbGroup.Status.GroupID { _, err = r.netbird.Networks.Routers(*nbrp.Status.NetworkID).Update(ctx, routers[0].Id, api.NetworkRouterRequest{ Enabled: true, @@ -317,6 +324,7 @@ func (r *NBRoutingPeerReconciler) handleRouter(ctx context.Context, nbrp *netbir return nil } +// handleSetupKey reconcile setup key and regenerate if invalid func (r *NBRoutingPeerReconciler) handleSetupKey(ctx context.Context, req ctrl.Request, nbrp *netbirdiov1.NBRoutingPeer, nbGroup netbirdiov1.NBGroup, logger logr.Logger) (*ctrl.Result, error) { networkName := r.ClusterName if r.NamespacedNetworks { @@ -381,10 +389,11 @@ func (r *NBRoutingPeerReconciler) handleSetupKey(ctx context.Context, req ctrl.R if (err != nil && strings.Contains(err.Error(), "not found")) || setupKey.Revoked { nbrp.Status.SetupKeyID = nil + // Requeue to avoid repeating code return &ctrl.Result{Requeue: true}, nil } - // Check if valid setup key exists + // Check if secret is valid skSecret := corev1.Secret{} err = r.Client.Get(ctx, req.NamespacedName, &skSecret) if err != nil && !errors.IsNotFound(err) { @@ -415,6 +424,7 @@ func (r *NBRoutingPeerReconciler) handleSetupKey(ctx context.Context, req ctrl.R return nil, nil } +// handleGroup creates/updates NBGroup for routing peer func (r *NBRoutingPeerReconciler) handleGroup(ctx context.Context, req ctrl.Request, nbrp *netbirdiov1.NBRoutingPeer, logger logr.Logger) (*netbirdiov1.NBGroup, *ctrl.Result, error) { networkName := r.ClusterName if r.NamespacedNetworks { @@ -471,6 +481,7 @@ func (r *NBRoutingPeerReconciler) handleGroup(ctx context.Context, req ctrl.Requ return &nbGroup, nil, nil } +// handleNetwork Create/Update NetBird Network func (r *NBRoutingPeerReconciler) handleNetwork(ctx context.Context, req ctrl.Request, nbrp *netbirdiov1.NBRoutingPeer, logger logr.Logger) error { networkName := r.ClusterName if r.NamespacedNetworks { diff --git a/internal/controller/service_controller.go b/internal/controller/service_controller.go index 247f063..0285275 100644 --- a/internal/controller/service_controller.go +++ b/internal/controller/service_controller.go @@ -70,6 +70,7 @@ func (r *ServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct return r.hideService(ctx, req, svc, logger) } +// hideService deletes NBResource for Service func (r *ServiceReconciler) hideService(ctx context.Context, req ctrl.Request, svc corev1.Service, logger logr.Logger) (ctrl.Result, error) { var nbResource netbirdiov1.NBResource err := r.Client.Get(ctx, req.NamespacedName, &nbResource) @@ -98,6 +99,7 @@ func (r *ServiceReconciler) hideService(ctx context.Context, req ctrl.Request, s return ctrl.Result{}, nil } +// exposeService creates/updates NBResource for Service func (r *ServiceReconciler) exposeService(ctx context.Context, req ctrl.Request, svc corev1.Service, logger logr.Logger) (ctrl.Result, error) { routerNamespace := r.ControllerNamespace if r.NamespacedNetworks { @@ -177,6 +179,7 @@ func (r *ServiceReconciler) exposeService(ctx context.Context, req ctrl.Request, return ctrl.Result{}, nil } +// reconcileNBResource ensures NBResource settings are in-line with Service definition and annotations func (r *ServiceReconciler) reconcileNBResource(nbResource *netbirdiov1.NBResource, req ctrl.Request, svc corev1.Service, routingPeer netbirdiov1.NBRoutingPeer) error { groups := []string{fmt.Sprintf("%s-%s-%s", r.ClusterName, req.Namespace, req.Name)} if v, ok := svc.Annotations[serviceGroupsAnnotation]; ok { @@ -242,6 +245,7 @@ func (r *ServiceReconciler) reconcileNBResource(nbResource *netbirdiov1.NBResour } } } + // TODO: Handle removed policy name return nil } From 61af2cd508425ed8bb58d007847b1422b157ebc7 Mon Sep 17 00:00:00 2001 From: Maycon Santos Date: Wed, 5 Mar 2025 15:01:46 +0100 Subject: [PATCH 2/4] update wording, links and adding examples --- README.md | 17 +++++----- docs/usage.md | 90 +++++++++++++++++++++++++++++++++++++++------------ 2 files changed, 78 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 1c34577..7340909 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ https://github.com/user-attachments/assets/5472a499-e63d-4301-a513-ad84cfe5ca7b ## Description -This operator enables easily provisioning NetBird access on kubernetes clusters, allowing users to access internal resources directly. +This operator easily provides NetBird access on Kubernetes clusters, allowing users to access internal resources directly. ## Getting Started @@ -18,7 +18,7 @@ This operator enables easily provisioning NetBird access on kubernetes clusters, ### Deployment > [!NOTE] -> Helm Installation method is recommended due to automation of multiple settings within the deployment. +> Helm Installation method is recommended due to the automation of multiple settings within the deployment. #### Using Helm @@ -27,14 +27,15 @@ This operator enables easily provisioning NetBird access on kubernetes clusters, helm repo add netbirdio https://netbirdio.github.io/kubernetes-operator ``` 2. (Recommended) Install [cert-manager](https://cert-manager.io/docs/installation/#default-static-install). -1. (Recommended) Create a values.yaml file, check `helm show values netbirdio/kubernetes-operator` for more info. -1. Install using `helm install --create-namespace -f values.yaml -n netbird netbird-operator netbirdio/kubernetes-operator`. +3. (Recommended) Create a values.yaml file, check `helm show values netbirdio/kubernetes-operator` for more info. +4. Install using `helm install --create-namespace -f values.yaml -n netbird netbird-operator netbirdio/kubernetes-operator`. +> Learn more about the values.yaml options [here](helm/netbird-operator/values.yaml) and [Granting controller access to NetBird Management](docs/values.md#granting-controller-access-to-netbird-management). #### Using install.yaml > [!IMPORTANT] -> install.yaml only includes a very basic template for deploying a stripped down version of kubernetes-operator. -> This excludes any and all configuration for ingress capabilities, and requires cert-manager to be installed. +> install.yaml only includes a very basic template for deploying a stripped-down version of Kubernetes-operator. +> This excludes any and all configurations for ingress capabilities and requires the cert-manager to be installed. ```sh kubectl create namespace netbird @@ -43,13 +44,13 @@ kubectl apply -n netbird -f https://raw.githubusercontent.com/netbirdio/kubernet ### Usage -Checks [usage.md](docs/usage.md). +Check the usage of [usage.md](docs/usage.md) and examples. ## Contributing ### Prerequisites -To be able to develop on this project, you need to have the following tools installed: +To be able to develop this project, you need to have the following tools installed: - [Git](https://git-scm.com/). - [Make](https://www.gnu.org/software/make/). diff --git a/docs/usage.md b/docs/usage.md index 8e68da6..229318d 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -47,41 +47,89 @@ spec: ### Granting controller access to NetBird Management > [!IMPORTANT] -> NetBird kubernetes operator generates configurations using NetBird API, editing or deleting these configurations in the NetBird console may cause temporary network disconnection until the operator reconciles the configuration. +> The NetBird Kubernetes operator generates configurations using NetBird API; editing or deleting these configurations in the NetBird console may cause temporary network disconnection until the operator reconciles the configuration. 1. Create a Service User on your NetBird dashboard (Must be Admin). [Doc](https://docs.netbird.io/how-to/access-netbird-public-api#creating-a-service-user). -1. Create access token for the Service User (Must be Admin). [Doc](https://docs.netbird.io/how-to/access-netbird-public-api#creating-a-service-user). +1. Create an access token for the Service User (Must be Admin). [Doc](https://docs.netbird.io/how-to/access-netbird-public-api#creating-a-service-user). 1. Add access token to your helm values file under `netbirdAPI.key`. 1. Alternatively, provision secret in the same namespace as the operator and set the key `NB_API_KEY` to the access token generated. 1. Set `netbirdAPI.keyFromSecret` to the name of the secret created. 1. Set `ingress.enabled` to `true`. - 1. Optionally, to provision network immediately, set `ingress.router.enabled` to `true`. - 1. Optionally, to provision 1 network per kubernetes namespace, set `ingress.namespacedNetworks` to `true`. + 1. Optionally, to provision the network immediately, set `ingress.router.enabled` to `true`. + 1. Optionally, to provision 1 network per > The NetBird Kubernetes operator generates configurations using NetBird API; editing or deleting these configurations in the NetBird console may cause temporary network disconnection until the operator reconciles the configuration. namespace, set `ingress.namespacedNetworks` to `true`. 1. Run `helm install` or `helm upgrade`. +Minimum values.yaml example: +```yaml +netbirdAPI: + key: "nbp_XXxxxxxxXXXXxxxxXXXXXxxx" + +ingress: + enabled: true +``` + ### Exposing a Service > [!IMPORTANT] -> Ingress DNS Resolution requires DNS Wildcard Routing to be enabled, and at least one DNS Nameserver configured for clients. +> Ingress DNS Resolution requires DNS Wildcard Routing to be enabled and at least one DNS Nameserver configured for clients. |Annotation|Description|Default|Valid Values| |---|---|---|---| -|`netbird.io/expose`|Expose service using NetBird Network Resource||(`null`, `true`)| -|`netbird.io/groups`|Comma-separated list of group names to assign to Network Resource|`{ClusterName}-{Namespace}-{Service}`|Any comma-separated list of strings.| -|`netbird.io/resource-name`|Network Resource name|`{Namespace}-{Service}`|Any valid network resource name, make sure they're unique!| -|`netbird.io/policy`|Name of NBPolicy to propagate service ports as destination.||Name of any NBPolicy resource| -|`netbird.io/policy-ports`|Narrow down exposed ports in policy, leave empty for all ports.||Comma-separated integer list, integers must be between 0-65535| -|`netbird.io/policy-protocol`|Narrow down protocol for use in policy, leave empty for all protocols.||(`tcp`,`udp`)| +|`netbird.io/expose`| Expose service using NetBird Network Resource ||(`null`, `true`)| +|`netbird.io/groups`| Comma-separated list of group names to assign to Network Resource |`{ClusterName}-{Namespace}-{Service}`|Any comma-separated list of strings.| +|`netbird.io/resource-name`| Network Resource name |`{Namespace}-{Service}`|Any valid network resource name, make sure they're unique!| +|`netbird.io/policy`| Name of NBPolicy to propagate service ports as destination. ||Name of any NBPolicy resource| +|`netbird.io/policy-ports`| Narrow down exposed ports in a policy. Leave empty for all ports. ||Comma-separated integer list, integers must be between 0-65535| +|`netbird.io/policy-protocol`| Narrow down protocol for use in a policy. Leave empty for all protocols. ||(`tcp`,`udp`)| + +Example service: +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + replicas: 0 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:latest + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx-service + annotations: + netbird.io/expose: "true" + netbird.io/groups: "groupA,groupB" +spec: + selector: + app: nginx + ports: + - protocol: TCP + port: 8080 + targetPort: 80 + type: ClusterIP +``` ### Notes -* `netbird.io/expose` will interpret any string as true value, the only false value is `null`. +* `netbird.io/expose` will interpret any string as a `true` value; the only `false` value is `null`. * The operator does **not** handle duplicate resource names within the same network, it is up to you to ensure resource names are unique within the same network. * While the NetBird console will allow group names to contain commas, this is not allowed in `netbird.io/groups` annotation as commas are used as separators. * If a group already exists on NetBird console with the same name, NetBird Operator will use that group ID instead of creating a new group. * NetBird Operator will attempt to clean up any resources created, including groups created for resources. - * In case a group is used by resources that cannot be cleaned up by the operator, the operator will eventually ignore the group in NetBird. - * It's recommended to use unique groups per NetBird Operator installation to remove any possible conflicts. -* NetBird Operator does not validate service annotations on update, as this may cause unnecessary overhead on any Service update. + * If a group is used by resources that the operator cannot clean up, the operator will eventually ignore the group in NetBird. + * It's recommended that unique groups be used per NetBird Operator installation to remove any possible conflicts. +* The Operator does not validate service annotations on updates, as this may cause unnecessary overhead on any Service update. ### Managing Policies @@ -102,11 +150,11 @@ ingress: - udp bidirectional: true # Optional, defaults to true ``` -2. Reference policy in Services using `netbird.io/policy=default`, this will add relevant ports and destination groups to Policy. -3. (Optional) limit specific ports in exposed service by adding `netbird.io/policy-ports=443`. -4. (Optional) limit specific protocol in exposed service by adding `netbird.io/policy-protocol=tcp`. +2. Reference policy in Services using `netbird.io/policy=default`, this will add relevant ports and destination groups to policy. +3. (Optional) Limit specific ports in exposed service by adding `netbird.io/policy-ports=443`. +4. (Optional) Limit specific protocol in exposed service by adding `netbird.io/policy-protocol=tcp`. #### Notes -* Each NBPolicy will only create policies in NetBird console when information provided is enough to create one. If there are no services acting as destination, or specified services do not conform to the protocol(s) defined, policy will not be created. -* Each NBPolicy will create one Policy in NetBird console per protocol specified as long as protocol has destinations, this ensures better secured policies by separating ports for TCP and UDP. -* Policies currently do not support ICMP protocol, as ICMP is not supported in kubernetes services, and there are [no current plans to support it](https://discuss.kubernetes.io/t/icmp-support-for-kubernetes-service/21738). +* Each NBPolicy will only create policies in the NetBird console when the information provided is enough to create one. If no services act as a destination or specified services do not conform to the protocol(s) defined, the policy will not be created. +* Each NBPolicy will create one policy in the NetBird console per protocol specified as long as the protocol has destinations; this ensures better-secured policies by separating ports for TCP and UDP. +* Policies currently do not support ICMP protocol, as ICMP is not supported in Kubernetes services, and there are [no current plans to support it](https://discuss.kubernetes.io/t/icmp-support-for-kubernetes-service/21738). From 547f285e47e70bc423cd96402d3e22747b85adfe Mon Sep 17 00:00:00 2001 From: Maycon Santos Date: Wed, 5 Mar 2025 15:03:33 +0100 Subject: [PATCH 3/4] fix link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7340909..dcf2029 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ helm repo add netbirdio https://netbirdio.github.io/kubernetes-operator 3. (Recommended) Create a values.yaml file, check `helm show values netbirdio/kubernetes-operator` for more info. 4. Install using `helm install --create-namespace -f values.yaml -n netbird netbird-operator netbirdio/kubernetes-operator`. -> Learn more about the values.yaml options [here](helm/netbird-operator/values.yaml) and [Granting controller access to NetBird Management](docs/values.md#granting-controller-access-to-netbird-management). +> Learn more about the values.yaml options [here](helm/kubernetes-operator/values.yaml) and [Granting controller access to NetBird Management](docs/usage.md#granting-controller-access-to-netbird-management). #### Using install.yaml > [!IMPORTANT] From 58c283116a4d1327a59a2196310040ae772a000c Mon Sep 17 00:00:00 2001 From: Maycon Santos Date: Wed, 5 Mar 2025 15:05:02 +0100 Subject: [PATCH 4/4] add cluster name --- docs/usage.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/usage.md b/docs/usage.md index 229318d..84ff0a7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -66,6 +66,8 @@ netbirdAPI: ingress: enabled: true +cluster: + name: kubernetes ``` ### Exposing a Service