ADR: Architecture document

[viniciusdc]

Important

This is a Work in Progress repository and task; thus, expect descriptions, deliverables, and overall decisions to change before, during, and after the development of its MVP.

Nebari Infrastructure Core (NIC) is a standalone Go CLI tool that manages cloud infrastructure for Nebari, using HCL and declarative configuration semantics to easily deploy all required components for a stable Kubernetes cluster, supporting multi-cloud architectures. NIC will later host an internal ecosystem of supported apps, managed internally by ArgoCD (see ArgoCD core concepts), referred to throughout this as "Apps".

Before introducing nic-operator, it is important to clarify what a Kubernetes Operator is and why it fits this problem space. In short, an operator is an application that runs inside a Kubernetes cluster and encodes operational knowledge for managing a specific domain. It typically introduces a Custom Resource Definition (CRD) that represents the desired state for some capability (e.g., “this app should be exposed at this hostname with TLS and SSO”), and a controller that continuously reconciles the actual cluster resources until they match that desired state. This “control loop” model allows platform behavior to be expressed declaratively and made repeatable, auditable, and self-healing (see: Kubernetes Operators — what are they?).

We are building nic-operator to standardize and automate “platform wiring” for applications in the NIC Kubernetes ecosystem. The goal is to standardize, in an automated fashion, all necessary hand edges of an app deployment alongside NIC’s "foundational software" layer (discussed in more detail at nebari-dev/nebari-infrastructure-core#5). Right now, we can assume the following software pieces represent the foundational layer of NIC:

• Envoy Gateway (Gateway API) for ingress routing
• cert-manager for TLS certificates
• Keycloak for SSO/OIDC authentication and user management
• Istio for mesh capabilities (optional integration, not required for MVP)
• Argo CD for GitOps-based application deployment (Helm)

There is, of course, an additional internal layer dedicated to metric collection and telemetry, but that will be considered a stretch goal for this initial operator implementation. But it should be kept in mind for future talks.

This operator’s primary objective is, in principle, self-service onboarding of NIC's platform "apps".

When a new app is deployed via Helm/Argo CD, the app declares minimal intent, and the platform automatically configures routing, TLS, and SSO — consistently and securely — without manual per-app ingress/cert/auth configuration.

To implement nic-operator using a conventional, maintainable Kubernetes controller layout, we will use the Kubebuilder framework. Kubebuilder provides scaffolding and patterns for defining CRDs (OpenAPI schemas), generating manifests, and implementing reconciliation logic, all aligned with Kubernetes API conventions and controller-runtime best practices.

Why Kubebuilder as a framework?

I am choosing Kubebuilder (Go + controller-runtime) as the primary framework for nic-operator because it matches NIC’s Go-first direction and keeps the operator as “just Kubernetes”: CRDs + controllers + reconciliation loops, following upstream conventions. For reference, see Kubebuilder’s Quick Start and the broader operator pattern overview from CNCF Kubernetes Operators: what are they?.

• Kubernetes-native API design for our onboarding contract

  • Our core deliverable is a stable CRD contract (e.g., NicApp) with strong schema validation, versioning conventions, and a first-class status subresource (conditions, observed generation). Kubebuilder is built precisely for designing and implementing Kubernetes APIs.
  • nic-operator will reconcile across multiple API groups (Gateway API resources like HTTPRoute, cert-manager resources, Envoy Gateway policy CRDs, plus Secrets/status). Go/controller-runtime makes it easier to keep this logic strongly typed and testable at scale.
  • Because Kubebuilder uses the standard RBAC/codegen pipeline, it’s straightforward to evolve RBAC with the controller as we split reconcilers by domain (routing/tls/auth/keycloak) and/or introduce admission validation later.

We also considered Operator SDK (Go), Kopf (python) and Juju (charms). Both are valid ecosystems, but they introduce additional scope and/or supporting dependencies that are larger than what we need for an internal, Argo CD–driven operator. For that reason, they were not selected for the MVP path. Specifically, the SDK path was drafted out because it relies on the OLM (Operator Lifecycle Manager) ecosystem, which, for this single-operator structure, does not make sense.

Note

We should reconsider this depending on any later decisions about relying on other operators to handle the creation of Grafana (or the LGTM stack), cert-manager, Istio, etc.

TLDR

This issue delivers the primary architecture Architectural Decision Record (ADR) that defines the onboarding contract (CRD-first vs annotations), domain boundaries (routing/tls/auth/keycloak/core), ownership/GC rules to avoid Argo CD conflicts, security boundaries (RBAC + secrets), status/readiness expectations, and a phased roadmap.

Additional considerations

Additionally, for early development/testing, we intend to mirror the local cluster bootstrap workflow/structure used by the vor-infra repository as a template idea to stand up a cluster with the required platform services.

[viniciusdc]

Below are a few considerations and assumptions that will be "needed" later, but will end up being too under the hood to be easily referenced, so I am outlining here for my own reference.

RBAC and Least-Privilege (practical)

Even if nic-operator is installed cluster-wide, it must behave as if it only manages namespaces that explicitly opt in to NIC app onboarding.

In practice, “least privilege” here has two parts:

1. RBAC capabilities (what the operator is allowed to do)
2. Policy enforced by reconciliation (where the operator will actually do it)

At a minimum, the operator should be limited to:

• Read NicApp CRDs cluster-wide (or only in opted-in namespaces)
• Write generated resources only in opted-in namespaces
• Read secrets referenced by NicApp only in the same namespace
• Never read secrets cross-namespace

There are two reasonable deployment models:

• Recommended (closest to least-privilege):

  • nic-operator can watch NicApp cluster-wide, but it only receives write permissions through RoleBindings in namespaces labeled nic.io/managed=true.
  • This avoids giving the operator blanket write access across all namespaces, and matches a multi-tenant/GitOps posture.

• Simpler (looser RBAC, stricter code policy):

  • nic-operator has cluster-wide write permissions for the resources it manages, but it must refuse to reconcile outside opted-in namespaces.
  • This is easier operationally, but it should be treated as a stepping stone, and later hardened (e.g., admission policies).

A special case is secrets: RBAC cannot easily restrict the operator to “only the secrets referenced by NicApp” without per-app RoleBindings or resourceNames. For the MVP we will adopt a strict convention instead:

• NicApp may only reference a Secret in the same namespace
• the operator will use get (not list) for Secrets where possible
• the operator will validate the required keys are present (client_id, client_secret, etc.)
• future hardening can add a validating webhook or policy engine (Kyverno/Gatekeeper) to enforce these constraints before reconciliation.

---

Domain boundaries and how they map to permissions

To keep the implementation maintainable, responsibilities are split by domain. Each domain implies a specific set of Kubernetes API interactions, and RBAC should follow that split:

• core/

  • validation, naming, status/conditions, events
  • needs: watch NicApp, update status, emit events

• routing/

  • generates HTTPRoute (Gateway API) and attaches it to the proper shared Gateway (public vs internal)
  • needs: manage HTTPRoute, read shared Gateway objects

• tls/

  • integrates with cert-manager following the chosen TLS strategy (wildcard vs per-host)
  • needs: read TLS secrets; optionally manage Certificate objects (per-host mode)

• auth/

  • configures edge auth enforcement via Envoy Gateway policy CRDs
  • needs: manage auth policy CRDs; read OIDC client secret (same namespace)

• keycloak/ (Phase 2)

  • provisions Keycloak clients and syncs credentials into Kubernetes
  • needs: read Keycloak admin credentials (if used), write OIDC client secret in namespace, network access to Keycloak

This split also makes it possible to later separate controllers / service accounts if we want even stronger least-privilege.

---

Flow (domains + resource ownership)

flowchart TB
  subgraph GitOps["GitOps"]
    Argo["Argo CD deploys Helm charts"]
    Chart["App chart: Deployment/Service + NicApp"]
  end
  Argo --> Chart

  subgraph Cluster["Kubernetes cluster"]
    direction TB

    subgraph NS["Opted-in namespace (nic.io/managed=true)"]
      AppCR["NicApp"]
      Svc["Service"]
      Dep["Deployment"]
      Secret["Secret: OIDC client credentials"]
      Route["HTTPRoute"]
      Policy["Envoy Gateway auth policy"]
      Cert["Certificate (optional)"]
    end

    subgraph Shared["Shared platform resources"]
      GWpub["Gateway: public"]
      GWin["Gateway: internal"]
      Issuer["Issuer/ClusterIssuer"]
      KC["Keycloak"]
    end

    subgraph Op["nic-operator"]
      Core["core"]
      Routing["routing"]
      TLS["tls"]
      Auth["auth"]
      KProv["keycloak (phase 2)"]
    end
  end

  Chart --> Dep
  Chart --> Svc
  Chart --> AppCR

  AppCR --> Core
  Core --> Routing --> Route --> GWpub
  Route --> GWin

  Core --> TLS --> Cert --> Issuer
  TLS --> GWpub
  TLS --> GWin

  Core --> Auth --> Policy --> GWpub
  Policy --> GWin
  Auth --> Secret
  Policy --> KC

  Core -. optional .-> KProv
  KProv -. phase 2 .-> KC
  KProv -. phase 2 .-> Secret

Loading

[viniciusdc]

The expected layout for the CRD configuration of a deployed app:

apiVersion: apps.nic.dev/v1
kind: NicApp
metadata:
  name: nginx-demo
  namespace: demo-app
  labels:
    app.kubernetes.io/name: nginx-demo
    app.kubernetes.io/managed-by: kustomize
spec:
  # The hostname where the application will be accessible
  hostname: nginx-demo.nic.local

  # Reference to the Kubernetes Service
  service:
    name: nginx-demo
    port: 80

  # TLS configuration (optional, defaults shown)
  tls:
    enabled: true
    mode: wildcard  # Use shared wildcard certificate

  # Authentication configuration (optional)
  auth:
    enabled: false 
    # To enable auth:
    # enabled: true
    # provider: keycloak
    # scopes:
    #   - openid
    # provisionClient: true

  gateway: public

[viniciusdc]

As suggested in the latest meeting, we should use NebariApp instead of NicApp, since it will handle Nebari apps as part of the Nebari OS ecosystem.

[viniciusdc]

rename nic-operator to nebari-operator

[viniciusdc]

Separation of Concerns: The operator manages HTTPRoutes only. TLS certificates and Gateway configuration are managed by the foundational infrastructure (cert-manager + Envoy Gateway).

Routing Configuration: Simplified TLS configuration to a boolean flag that controls which Gateway listener (HTTP/HTTPS) the HTTPRoute references, rather than managing certificates.