feat: single-replica ovn-central + PVC + LoadBalancer + Kamaji split-cluster deployment#6793
Open
oilbeater wants to merge 20 commits into
Open
feat: single-replica ovn-central + PVC + LoadBalancer + Kamaji split-cluster deployment#6793oilbeater wants to merge 20 commits into
oilbeater wants to merge 20 commits into
Conversation
Introduce OVN_CENTRAL_MODE=single in the Helm chart and rework the chart templates, startup scripts and leader-checker so ovn-central can run as a single-replica Deployment with the OVN DB on a PVC. The pod can drift to another node on host failure as long as the StorageClass supports cross-node attach. - chart: new OVN_CENTRAL_MODE value and ovn-central.storage block; ovnCentralReplicas / ovnCentralNodeIPs helpers; new central-pvc.yaml manifest; central-deploy.yaml branches replicas, strategy, podAntiAffinity and the host-config-ovn volume; the three ovn-nb / ovn-sb / ovn-northd Services drop the leader-label selector in single mode so the lone pod is always the endpoint; controller-deploy.yaml and ovsovn-ds.yaml route OVN_DB_IPS through the new helper so clients fall back to the Service ClusterIPs. - start-db.sh: the existing NODE_IPS="" standalone path now also gets --ovn-northd-n-threads honoured and cleans stale sockets/pids so a drifted pod can start cleanly on a new node. - ovn-is-leader.sh / ovn-healthcheck.sh: short-circuit raft-only logic in single mode (no cluster/status query, no ovn_northd lock steal); keep DB storage-status check + compaction. - pkg/ovn_leader_checker: filter empty entries out of remoteAddresses and add a single-replica branch that just keeps the pod labelled as leader for nb/sb/northd and runs compaction. Default OVN_CENTRAL_MODE remains cluster; `helm template` of the cluster mode renders identically to before. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…e script Wire the single-replica deployment mode into install.sh so users who do not use Helm can opt in via ENABLE_SINGLE_REPLICA_OVN=true with optional OVN_CENTRAL_STORAGE_CLASS / OVN_CENTRAL_PVC_SIZE overrides. - install.sh: when ENABLE_SINGLE_REPLICA_OVN=true, override addresses="" / count=1 so the rest of the script naturally emits an empty NODE_IPS / OVN_DB_IPS. The ovn.yaml HEREDOC is parameterised via pre-computed YAML fragments (PVC block, leader-label selectors, Deployment strategy, podAntiAffinity, host-config-ovn volume), so the cluster-mode output is identical to before modulo one harmless empty line. - restore-ovn-nb-db.sh: add a `single <backup.db>` subcommand that scales ovn-central to 0, spins up a busybox helper pod mounting the same PVC, kubectl-cp's the backup in, swaps it into place (keeping the previous file as .bak.<ts>) and scales back up. The existing cluster mode is preserved unchanged behind `cluster` (default). Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
- docs/single-replica-deployment.md: user-facing guide covering when to pick the new mode, StorageClass requirements (cross-node attach, WaitForFirstConsumer), Helm and install.sh install paths, a failover drill, backup/restore via restore-ovn-nb-db.sh, migration between modes and known limitations. - test/e2e/single-replica: new ginkgo suite that auto-skips when the cluster is not in single-replica mode (detected via the ovn-nb Service selector) and otherwise verifies replicas=1 + Recreate + PVC volume, leaderless Service selectors, the pod's leader labels, basic subnet+pod networking, and recovery after the ovn-central pod is deleted. - makefiles: register the new e2e package in e2e-build, add kube-ovn-single-replica-e2e and a kind-install-single-replica shortcut that installs against kind's default `standard` StorageClass. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…r NodePort
Add a `service.{type,loadBalancerIP,externalTrafficPolicy}` block under
`ovn-central` in the chart values (and the matching `OVN_CENTRAL_SERVICE_TYPE`
/ `OVN_CENTRAL_LB_IP` / `OVN_CENTRAL_EXTERNAL_TRAFFIC_POLICY` envs in
install.sh). The three Services for ovn-nb / ovn-sb / ovn-northd now honour
the chosen type so the OVN DB can be reached from data-plane components
that live outside the management cluster — for example a Kamaji-style
topology where ovn-controller / kube-ovn-cni / kube-ovn-controller run in
a separate tenant cluster and need a stable VIP to connect back to.
Default stays ClusterIP, so cluster-mode helm rendering and the existing
install.sh output are unchanged.
docs: add an "Exposing the OVN DB to data planes outside the cluster"
section to docs/single-replica-deployment.md, covering Helm values,
install.sh envs, how tenant clusters point ovs-ovn / kube-ovn-cni at the
LB via OVN_DB_IPS, and a note that SSL should be enabled before exposing
the DB outside the cluster.
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
Introduce a single value, `installMode`, that lets the same Helm chart serve
three different scenarios from one source of truth:
- `full` (default): everything in one cluster, identical to today.
- `controlPlaneOnly`: only ovn-central + ovn-nb/ovn-sb/ovn-northd Services and
their RBAC. Use on the management cluster of a Kamaji-style split.
- `dataPlaneOnly`: CRDs + kube-ovn-controller + ovs-ovn + kube-ovn-cni +
pinger + monitor and their RBAC. Use on tenant data-plane clusters; the
agents and controller connect to the management cluster's exposed
ovn-nb / ovn-sb via the new `externalOvnCentral.endpoint` value.
Implementation:
- `values.yaml`: new `installMode` and `externalOvnCentral.{endpoint,nbPort,
sbPort}` keys.
- `_helpers.tpl`: new `kubeovn.renderControlPlane` / `kubeovn.renderDataPlane`
gates; `kubeovn.ovnCentralNodeIPs` now emits `externalOvnCentral.endpoint`
in dataPlaneOnly mode so kube-ovn-controller and ovs-ovn get
`OVN_DB_IPS=<lb-ip>` for free.
- Templates wrapped with the appropriate gate:
- controlPlaneOnly: central-deploy, central-pvc, nb-svc, sb-svc, northd-svc.
- dataPlaneOnly: kube-ovn-crd, controller-deploy/svc, monitor-deploy/svc,
ovsovn-ds, ovncni-ds/svc, pinger-ds/svc, ovn-dpdk-ds, ic-controller,
vpc-nat-config, post-delete-hook, pre-upgrade-ovs-ovn,
upgrade-ovs-ovn.
- Both planes: `ovn-ovs` ServiceAccount + `system:ovn-ovs` ClusterRole +
binding (used by ovn-central in mgmt and ovs-ovn in tenant), plus the
optional `kube-ovn-tls` Secret manifest.
- `hack/gen-crd.sh`: wrap the generated CRD bundle with the data-plane gate
when copying into the v1 chart so the wrapper survives regenerations.
Idempotent (re-running gen-crd does not stack wrappers).
- `docs/kamaji-deployment.md`: full walkthrough of the two-install pattern,
including LoadBalancer requirements, TLS Secret sync between clusters and
version-lockstep guidance.
Default `installMode=full` renders identically to before — verified via
`helm template` diff against the previous PR's output (zero diff).
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…ovs-ovn restart
Address three issues raised in code review of the Kamaji split + single-
replica work:
- externalOvnCentral.{nbPort,sbPort} were declared in values.yaml but never
reached the agents. start-controller.sh / start-ovs.sh hardcoded 6641 /
6642 when OVN_DB_IPS was set, so any LoadBalancer or NodePort install that
remapped ports could not connect. The scripts now read OVN_NB_PORT /
OVN_SB_PORT env vars (defaulting to the in-cluster Service ports), and the
chart wires the externalOvnCentral ports into both kube-ovn-controller and
ovs-ovn DaemonSet env via new kubeovn.ovnNbPort / kubeovn.ovnSbPort
helpers.
- The previous `ovn-central.storage.enabled` toggle could be flipped off
while the Deployment still mounted a PVC named "ovn-central-data", leaving
ovn-central permanently Pending. Replace the boolean with an explicit
`existingClaim` field: when empty (default) the chart creates the PVC;
when set, in-chart PVC creation is skipped and the Deployment mounts the
named claim instead. Removes the silent footgun and gives operators a
clean handoff to externally-managed storage.
- restore-ovn-nb-db.sh's single-mode flow ran `kubectl rollout restart ds
ovs-ovn` unconditionally. In a controlPlaneOnly management-cluster install
there is no ovs-ovn DaemonSet, so the restore exited non-zero after a
successful DB restore. Gate the restart on `kubectl get ds ovs-ovn` and
print a hint to restart ovs-ovn on the data-plane cluster instead.
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
oilbeater
changed the title
4 tasks
Coverage Report for CI Build 26581686879Warning Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes. Coverage increased (+0.2%) to 25.488%Details
Uncovered Changes
Coverage Regressions174 previously-covered lines in 1 file lost coverage.
Coverage Stats
💛 - Coveralls |
Second round of review fixes for the Kamaji split / single-replica work: - upgrade-ovs.sh and start-ovs-dpdk-v2.sh both still assume a local ovn-central (the former waits on `deploy/ovn-central`, the latter pulls OVN_SB_SERVICE_HOST from Kubernetes env injection). Rendering them in a dataPlaneOnly tenant install would crashloop or block upgrades. Add a new `kubeovn.renderFullOnly` helper and gate the two ovs-ovn upgrade hooks and the ovn-dpdk DaemonSet behind it. controlPlaneOnly installs naturally skip them already; dataPlaneOnly users who need DPDK or in-place OVS upgrades will need a follow-up that teaches those scripts to honor externalOvnCentral. - Tighten the docs/values comments around `externalOvnCentral.endpoint`: the OVN connection string format `tcp:[host]:port` is fragile against older OVN parsers when `host` is a DNS hostname, so recommend an IP literal (recent OVN 22.03+ tolerates hostnames). Note the DPDK / upgrade- hook follow-up explicitly in docs/kamaji-deployment.md. Note: ovs-healthcheck.sh's `gen_conn_str 6642` reference is dead code (only present inside a commented-out echo). The probe actually talks to local OVS sockets, so it does not need port remapping changes. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the kube-ovn Helm chart and supporting scripts to support Kamaji-style split-cluster deployments by (1) enabling a single-replica ovn-central mode backed by a PVC, (2) optionally exposing OVN DB Services via LoadBalancer/NodePort, and (3) introducing installMode to render control-plane-only vs data-plane-only subsets from the same chart.
Changes:
- Add
OVN_CENTRAL_MODE=singlewith PVC-backedovn-central(Recreate strategy, leader-less Service selectors, standalone-aware health/leader logic). - Add
installMode=full|controlPlaneOnly|dataPlaneOnlywith Helm render gates and CRD gating for tenant clusters. - Wire
externalOvnCentral.{endpoint,nbPort,sbPort}through chart helpers and scripts so external NB/SB port remaps work; add docs + a dedicated single-replica e2e suite/targets.
Reviewed changes
Copilot reviewed 33 out of 40 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/single-replica/single_replica_test.go | Adds e2e coverage for single-replica ovn-central + PVC + leader-less Services + recovery. |
| pkg/ovn_leader_checker/ovn.go | Skips raft-only leader checks in single-replica (no peers) mode; filters empty remote addresses. |
| makefiles/kind.mk | Adds a kind install target for single-replica mode. |
| makefiles/e2e.mk | Builds/runs new single-replica e2e suite. |
| hack/gen-crd.sh | Wraps v1 chart CRD bundle with a data-plane render gate for installMode. |
| docs/single-replica-deployment.md | Documents single-replica mode and external exposure patterns. |
| docs/kamaji-deployment.md | Documents split-cluster (management/control-plane vs tenant/data-plane) deployment flow. |
| dist/images/start-ovs.sh | Honors OVN_SB_PORT when generating SB connection strings for external ovn-central. |
| dist/images/start-db.sh | Adds standalone-mode cleanup for drifted pods; threads flag for northd restart. |
| dist/images/start-controller.sh | Honors OVN_NB_PORT/OVN_SB_PORT for external ovn-central connections. |
| dist/images/restore-ovn-nb-db.sh | Adds single restore mode that writes a standalone backup into the PVC; skips ovs-ovn restart when absent. |
| dist/images/ovn-is-leader.sh | Adds standalone-mode labeling + storage-status checks and compaction. |
| dist/images/ovn-healthcheck.sh | Adds standalone-mode storage-status readiness checks (no raft cluster/status). |
| dist/images/install.sh | Adds single-replica mode + Service exposure knobs; injects PVC + strategy/selector variants. |
| charts/kube-ovn/values.yaml | Introduces OVN_CENTRAL_MODE, installMode, externalOvnCentral, ovn-central storage/service settings. |
| charts/kube-ovn/templates/vpc-nat-config.yaml | Gates data-plane-only resource rendering via installMode. |
| charts/kube-ovn/templates/upgrade-ovs-ovn.yaml | Gates full-only upgrade hook rendering. |
| charts/kube-ovn/templates/sb-svc.yaml | Adds control-plane render gate; supports LB/NodePort; drops leader selector in single mode. |
| charts/kube-ovn/templates/pre-upgrade-ovs-ovn.yaml | Gates full-only pre-upgrade hook rendering. |
| charts/kube-ovn/templates/post-delete-hook.yaml | Gates data-plane render for post-delete hook resources. |
| charts/kube-ovn/templates/pinger-svc.yaml | Gates data-plane rendering for pinger Service. |
| charts/kube-ovn/templates/pinger-ds.yaml | Gates data-plane rendering for pinger DaemonSet. |
| charts/kube-ovn/templates/ovsovn-ds.yaml | Gates data-plane rendering; wires external ovn-central endpoint + SB port. |
| charts/kube-ovn/templates/ovncni-svc.yaml | Gates data-plane rendering for CNI Service. |
| charts/kube-ovn/templates/ovncni-ds.yaml | Gates data-plane rendering for CNI DaemonSet. |
| charts/kube-ovn/templates/ovn-sa.yaml | Partitions ServiceAccounts by installMode (data-plane vs both-planes). |
| charts/kube-ovn/templates/ovn-dpdk-ds.yaml | Gates full-only DPDK DaemonSet rendering. |
| charts/kube-ovn/templates/ovn-CRB.yaml | Partitions ClusterRoleBindings/RoleBindings by installMode. |
| charts/kube-ovn/templates/ovn-CR.yaml | Partitions ClusterRoles by installMode. |
| charts/kube-ovn/templates/northd-svc.yaml | Adds control-plane render gate; LB/NodePort support; drops leader selector in single mode. |
| charts/kube-ovn/templates/nb-svc.yaml | Adds control-plane render gate; LB/NodePort support; drops leader selector in single mode. |
| charts/kube-ovn/templates/monitor-svc.yaml | Gates data-plane rendering for monitor Service. |
| charts/kube-ovn/templates/monitor-deploy.yaml | Gates data-plane rendering for monitor Deployment. |
| charts/kube-ovn/templates/kube-ovn-crd.yaml | Gates CRDs to render only in data-plane installs. |
| charts/kube-ovn/templates/ic-controller-deploy.yaml | Gates IC controller to data-plane rendering. |
| charts/kube-ovn/templates/controller-svc.yaml | Gates controller Service to data-plane rendering. |
| charts/kube-ovn/templates/controller-deploy.yaml | Gates controller Deployment; wires external ovn-central endpoint + NB/SB ports. |
| charts/kube-ovn/templates/central-pvc.yaml | Adds PVC template for single-replica ovn-central with optional existingClaim. |
| charts/kube-ovn/templates/central-deploy.yaml | Adds control-plane render gate; single-mode replicas/strategy/affinity/volume branching. |
| charts/kube-ovn/templates/_helpers.tpl | Adds helpers for installMode gating, ovn-central replicas/endpoints, and external NB/SB ports. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+47
to
+54
| ovn-central: | ||
| storage: | ||
| enabled: true | ||
| storageClassName: my-csi # leave empty to use the cluster default | ||
| size: 10Gi | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| ``` |
Comment on lines
+118
to
+123
| ovn-central: | ||
| storage: | ||
| enabled: true | ||
| storageClassName: my-csi | ||
| size: 10Gi | ||
| service: |
Comment on lines
+160
to
+162
| A full Kamaji integration also needs a "data-plane only" rendering of the | ||
| chart that skips `ovn-central` and its PVC — that's tracked separately and | ||
| not part of this change. |
Comment on lines
+65
to
+70
| ovn-central: | ||
| storage: | ||
| enabled: true | ||
| storageClassName: my-csi | ||
| size: 10Gi | ||
| service: |
Comment on lines
+204
to
+208
| # Set existingClaim to skip in-chart PVC creation and point the | ||
| # Deployment at a PVC the operator manages out-of-band (e.g. via a | ||
| # storage operator). The Deployment always mounts claim "ovn-central-data" | ||
| # so the value here must equal "ovn-central-data" — leaving the field | ||
| # empty means the chart creates the PVC itself. |
… name Third round of review fixes for the Kamaji split: - `kubeovn.ovnCentralNodeIPs` now uses Helm's `required` to fail at template-time when `installMode=dataPlaneOnly` but `externalOvnCentral. endpoint` is empty. Without that guard, agents fell back to `OVN_NB_SERVICE_HOST` (which is not injected when the local ovn-nb Service is not rendered) and crashed under `set -u`. - `ic-controller-deploy.yaml` was still injecting `OVN_DB_IPS` from the tenant cluster's own master nodes, and `start-ic-controller.sh` was hardcoding ports 6641/6642. Route both through the new `kubeovn.ovnCentralNodeIPs` / `kubeovn.ovnNbPort` / `kubeovn.ovnSbPort` helpers and have the script honor `OVN_NB_PORT` / `OVN_SB_PORT`. With this, `installMode=dataPlaneOnly` + `ENABLE_IC=true` actually points the ic-controller at the management cluster's exposed OVN DB. - `restore-ovn-nb-db.sh single` discovers the PVC name from the ovn-central Deployment's `host-config-ovn` volume rather than hardcoding `ovn-central-data`. Operators using `ovn-central.storage.existingClaim=<custom-name>` can now run the bundled restore flow against the right PVC. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…talLB annotation Two more review fixes: - When `installMode=dataPlaneOnly` + `networking.ENABLE_SSL=true` the chart would self-sign a fresh kube-ovn-tls Secret on the tenant cluster, producing certs that the management cluster's ovn-central does not trust. Fail at template-time with a clear message telling the operator to sync the Secret from the management cluster first. - The three OVN Services share `ovn-central.service.loadBalancerIP` but cloud LB providers without VIP sharing assign each Service its own IP, breaking the assumption that one externalOvnCentral.endpoint can reach all three databases. Emit the MetalLB `allow-shared-ip: kube-ovn-central` annotation on all three Services when serviceType=LoadBalancer; this is a no-op on non-MetalLB providers but lets the common MetalLB case work without extra configuration. Document the constraint and the cloud-LB workaround (NodePort fronted by an external LB) in docs/kamaji-deployment.md. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…own on PVC
Three more review fixes for the combined PR:
- `dist/images/install.sh`: when `ENABLE_SINGLE_REPLICA_OVN=true` the
precompute step intentionally sets `addresses=""`, but the generated
ovn.yaml emitted unquoted `value: $addresses` for `NODE_IPS` and
`OVN_DB_IPS`. The empty expansion produces YAML `value:` (null), which
Kubernetes rejects for string-typed env vars and the apply step fails.
Wrap all five sites in `"$addresses"` so empty becomes `""`.
- `install.sh` also did not emit the MetalLB `allow-shared-ip` annotation
that the chart now adds in `OVN_CENTRAL_SERVICE_TYPE=LoadBalancer`
installs. Without it, only one of ovn-nb/sb/northd could claim the
same `loadBalancerIP`. Wire a new `${OVN_CENTRAL_SVC_ANNOTATIONS}`
precompute block into the three Service manifests.
- `charts/kube-ovn/templates/central-deploy.yaml`: the `hostpath-init`
init container `chown -R nobody: ... /etc/ovn ...` fails on PVC backends
that disallow chown (notably root-squashed NFS, which the docs
recommend for single-replica drift). In single-replica mode, run the
chown on `/var/run/ovn` and `/var/log/ovn` strictly and make the
`/etc/ovn` chown best-effort: if the backend refuses, log a hint and
continue. The main container runs as `nobody` and will create new
files with the correct owner regardless.
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
externalOvnCentral.endpoint is a single IP, so the three OVN Services (ovn-nb / ovn-sb / ovn-northd) must land on the same VIP for the tenant cluster to reach all of NB/SB/northd. With type=LoadBalancer but no explicit loadBalancerIP, cloud LB controllers allocate three different external IPs and the design silently breaks. Add a kubeovn.validateService helper that fails at template time when serviceType=LoadBalancer and loadBalancerIP is empty. Include it in the three svc templates. The fail message tells the operator to pick a VIP and notes that the MetalLB allow-shared-ip annotation is emitted automatically once they do. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…ener Two install.sh-side parity fixes matching what the Helm chart already does: - Reject OVN_CENTRAL_SERVICE_TYPE=LoadBalancer without OVN_CENTRAL_LB_IP. externalOvnCentral.endpoint is a single IP, so the three OVN Services must share one VIP; without an explicit IP cloud LB controllers allocate three different ones and tenants can only reach NB or SB. The chart already fails in this case via `kubeovn.validateService`; install.sh now matches. - In ENABLE_SINGLE_REPLICA_OVN mode, soften the hostpath-init container's chown of /etc/ovn. Some PVC backends (notably root-squashed NFS, which the docs recommend for cross-node drift) reject root chown, leaving the pod stuck in Init:CrashLoopBackOff. Run the chown on /var/run/ovn and /var/log/ovn strictly and make /etc/ovn best-effort, mirroring the branch already added to charts/kube-ovn/templates/central-deploy.yaml. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…elper Final round of review fixes: - Inject \`helm.sh/resource-policy: keep\` on every CRD so flipping \`installMode\` (e.g. \`full\` -> \`controlPlaneOnly\`) does not let Helm cascade-delete the live custom resources. The annotation is emitted by \`hack/gen-crd.sh\` when wrapping the bundle, so it survives every regeneration. - \`central-deploy.yaml\` now \`lookup\`s the existing ovn-central Deployment and fails the upgrade if its host-config-ovn volume type disagrees with the rendered \`OVN_CENTRAL_MODE\` (raft hostPath <-> single PVC). In-place mode switches would otherwise start ovn-central against an empty target and silently lose all live OVN state. Lookup returns empty during dry-run / template, so this only fires against real clusters. - \`restore-ovn-nb-db.sh\` single-mode helper pod now adds ovn-central's tolerations so it can schedule onto the same control-plane / master node that hosts the PVC; without them \`kubectl wait --for=condition=Ready\` would deadlock on single-replica installs whose only viable node is tainted. - The helper pod also chowns the restored DB to uid/gid 65534 (\`nobody\`) before exiting. \`kubectl cp\` extracts the tarball as the container's uid (root in busybox), and on PVC backends where the init-container chown is intentionally best-effort (root-squashed NFS), ovn-central came back as nobody unable to write its own DB. The chown is itself best-effort with a clear log line on backends that map root to nobody (NFS root_squash). Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
Two follow-ups from code review:
- kube-ovn-monitor reads ovn-central's local Unix sockets at
/run/ovn/ovn{nb,sb}_db.sock and the DB files at /etc/ovn/*.db, and
TryClientConnection exits the process after 5 retries. In
installMode=dataPlaneOnly there is no local ovn-central, so the
Deployment would crashloop on every tenant cluster. Move monitor-
deploy.yaml and monitor-svc.yaml to the renderFullOnly gate. The
kube-ovn-app SA / ClusterRole / RoleBinding stay in renderDataPlane
because kube-ovn-pinger also uses them. Add a Kamaji-doc note + a
follow-up in the limitations section.
- hack/gen-crd.sh injected a NEW `metadata.annotations:` block for each
CRD, which produced two annotations mappings per CRD. YAML loaders
keep one of those, so the helm.sh/resource-policy: keep annotation
was getting dropped during render. Merge the new key into the
existing controller-gen annotations block instead, so the rendered
YAML carries both keys on every CRD. Verified with yaml.safe_load on
the helm template output: 24/24 CRDs have keep: yes.
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…make restore image configurable Three follow-up review fixes: - Add helm.sh/resource-policy: keep to the ovn-central-data PVC. Without it, an installMode switch (e.g. controlPlaneOnly with OVN_CENTRAL_MODE=single -> dataPlaneOnly) drops the PVC out of the rendered manifest and Helm deletes the only copy of the standalone OVN DB. Same data-loss class as the central-deploy guard already added. - pkg/ovn_leader_checker classified a one-node raft cluster as "single-replica" because `remoteAddresses` collapses to an empty slice once the local address is filtered out, which silently skipped raft-header backup, isDBLeader queries, and ovn_northd lock stealing. Compute singleReplica from the raw --remoteAddresses flag (true only when it was passed explicitly empty) and store on Configuration. One-node raft installs now keep their full cluster-mode behaviour. - restore-ovn-nb-db.sh's single-mode helper pod hardcoded busybox:1.36, which breaks in air-gapped or private-registry environments. Read an optional RESTORE_HELPER_IMAGE env override and default to docker.io/library/busybox:1.36 so the operator can swap in any minimal Linux image they already have mirrored. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…l.sh mode switch Two final review fixes: - ovn-tls-secret.yaml previously rendered the Secret manifest in all SSL installs. In installMode=dataPlaneOnly the operator pre-seeds the Secret out-of-band (synced from the management cluster), so Helm tries to create or adopt it and fails with "resource already exists". Skip the manifest entirely in dataPlaneOnly mode and keep only the fail-fast guard that requires the pre-seeded Secret to exist before install. Other install modes still render the Secret (auto-generated CA in cluster mode, reused from existing in full mode). - install.sh now mirrors the chart's Deployment-level guard against in-place OVN_CENTRAL_MODE switches. It looks up the existing ovn-central Deployment's host-config-ovn volume and rejects flipping ENABLE_SINGLE_REPLICA_OVN if that would swap hostPath <-> PVC underneath a running release. Operators are told to back up, delete the existing Deployment, and re-run install.sh. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
Address Copilot review on PR #6793: - docs/single-replica-deployment.md (both Helm examples) and docs/kamaji-deployment.md (management-cluster values example): drop the obsolete `ovn-central.storage.enabled` field. That toggle was removed when existingClaim was introduced; the docs still referenced it, so any user copy-pasting would get a non-functional values file. - docs/single-replica-deployment.md "follow-up" paragraph claiming a data-plane-only chart rendering is tracked separately is no longer true — this PR ships it. Replace the paragraph with a pointer to docs/kamaji-deployment.md. - values.yaml comment on `ovn-central.storage.existingClaim` contradicted itself (claimed the value must be "ovn-central-data" while the Deployment template actually mounts whatever name is set). Reword to match the template behaviour: the field is the name of an external PVC; empty means the chart creates a default-named one. Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
Discovered while bringing up a real Kamaji-style local environment
(kind + Kamaji + tenant docker worker via kubeadm join). On the tenant
cluster, controller-deploy.yaml's `replicas: {{ include "kubeovn.
nodeCount" . }}` triggered the helper's `fail` because tenant nodes
don't carry the `kube-ovn/role=master` label (Kamaji moves the control
plane into the management cluster as pods).
Add a kubeovn.controllerReplicas helper:
- dataPlaneOnly: default to 1 (active/standby HA is by leader election,
not horizontal scale)
- everywhere else: keep nodeCount behaviour
- operators can override via `kube-ovn-controller.replicas`
controller-deploy.yaml now uses this helper. Verified end-to-end:
`helm install --kubeconfig=tenant-kc -f tenant-values.yaml` now
succeeds, the controller Deployment + ovs-ovn DS + kube-ovn-cni DS land
on the tenant worker, and the tenant worker's TCP path to the
management cluster's MetalLB VIP `172.18.255.210:6641/:6642` is open.
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
Turn the manual Kamaji-on-kind walkthrough into a reproducible automated e2e. The local validation covered the only flow that exercises both halves of installMode together (controlPlaneOnly + dataPlaneOnly + externalOvnCentral.endpoint), and CI/contributors need the same path. - hack/kamaji-e2e.sh: subcommands `setup`, `teardown`, `kubeconfig`, `vars`. Brings up the mgmt kind cluster, installs cert-manager, MetalLB and Kamaji, installs kube-ovn `installMode=controlPlaneOnly` with single-replica + LoadBalancer + storage, creates a TenantControlPlane via Kamaji, spawns a privileged docker container for the tenant worker (with containerd's native snapshotter to dodge nested-overlayfs whiteout issues), pre-pulls the kube-ovn image via a local registry, joins the worker via kubeadm and installs `installMode=dataPlaneOnly` pointed at the management cluster's MetalLB VIP. - test/e2e/kamaji/kamaji_test.go: a small Ginkgo suite that auto-skips when not running against a dataPlaneOnly tenant cluster. It verifies: (1) kube-ovn-controller defaults to replicas=1 (covers the kubeovn.controllerReplicas helper); (2) ovs-ovn really keeps an ESTAB connection to the external ovn-sb VIP via `ss` exec; (3) tenant Pods get IPs from the OVN subnet through the cross-cluster control path; (4) leader-election Lease lives on the tenant apiserver. - makefiles/kind.mk: new `kind-install-kamaji` / `kind-clean-kamaji` targets that wrap the shell script. - makefiles/e2e.mk: register `./test/e2e/kamaji` in `e2e-build` and add `kube-ovn-kamaji-e2e` target. The target sources KUBECONFIG and the KUBE_OVN_KAMAJI_MGMT_VIP env from `hack/kamaji-e2e.sh vars` so the Ginkgo suite can identify the external VIP without rediscovering it. Reproducing locally: make build-dev make kind-install-kamaji make kube-ovn-kamaji-e2e make kind-clean-kamaji Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
…lash
CI is failing on every job at the kube-ovn-cni daemonset rollout. Root
cause: Kubernetes auto-injects environment variables for in-namespace
Services using the pattern `<SVCNAME>_PORT=tcp://<ClusterIP>:<port>`.
The Service named `ovn-nb` therefore implicitly creates
`OVN_NB_PORT=tcp://10.x.x.x:6641` in every kube-system pod's env, which
collided with the new variable I introduced in this PR for the
external-endpoint port. kube-ovn-controller then built a garbage
connection string like
--ovn-nb-addr=tcp:[172.18.0.3]:tcp://10.110.227.251:6641
so the OVN agents never came up.
Rename the variable everywhere to `KUBE_OVN_NB_PORT` /
`KUBE_OVN_SB_PORT` — the kube-ovn project prefix is unique and avoids
the kubelet's auto-injected names. Touch points:
- dist/images/start-controller.sh
- dist/images/start-ic-controller.sh
- dist/images/start-ovs.sh
- charts/kube-ovn/templates/controller-deploy.yaml
- charts/kube-ovn/templates/ic-controller-deploy.yaml
- charts/kube-ovn/templates/ovsovn-ds.yaml
The Helm-internal helper names (`kubeovn.ovnNbPort` /
`kubeovn.ovnSbPort`) stay unchanged; only the env vars they generate
into pods got renamed. The defaults (6641/6642) still mean
in-cluster Service mode for non-Kamaji installs.
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
End goal: deploy kube-ovn on a Kamaji-style split-cluster topology.
This PR replaces #6782 (single-replica + LoadBalancer exposure) and bundles
the Kamaji
installModework originally split out, plus three follow-upfixes from the first round of Codex review. Reviewing as one PR is easier
than coordinating two — the features are useless individually for the
target deployment shape.
What you get
OVN_CENTRAL_MODE=single— ovn-central as a single-replica Deploymentbacked by a PVC. Pod can drift to another node on host failure (provided
the StorageClass supports cross-node attach). New
central-pvc.yaml,central-deploy.yamlstrategy/affinity/volume branching, leaderlessService selectors,
start-db.shstandalone enhancements,pkg/ovn_leader_checkershort-circuits raft queries.ovn-central.service.type=LoadBalancer\|NodePort— expose the threeOVN DB Services for external clients (e.g. tenant clusters).
installMode=full\|controlPlaneOnly\|dataPlaneOnly— same chart servesthree deployment shapes from one source of truth. Default
fullrendersidentically to master (verified by
helm templatediff).externalOvnCentral.{endpoint,nbPort,sbPort}— tenant data-planeclusters point at the management cluster's exposed ovn-nb / ovn-sb via a
LoadBalancer / NodePort VIP, with full port-remap support.
How it composes for Kamaji
Full walkthrough including SSL Secret sync, version lockstep, and a
failover drill is in
docs/kamaji-deployment.md(with thesingle-replica details in
docs/single-replica-deployment.md).Codex review fixes (last commit)
externalOvnCentral.{nbPort,sbPort}: previously declared butignored —
start-controller.sh/start-ovs.shhardcoded 6641 / 6642when
OVN_DB_IPSwas set, so any LoadBalancer/NodePort that remappedports could not connect. Scripts now read
OVN_NB_PORT/OVN_SB_PORTenv vars wired through new chart helpers.
ovn-central.storage.enabledfootgun: replace with explicitexistingClaimso toggling away from in-chart PVC creation doesn'tleave the Deployment Pending forever.
ovs-ovnrestart in restore-ovn-nb-db.sh: skip thekubectl rollout restart ds ovs-ovnstep oncontrolPlaneOnlyinstalls (no DS there) — print a hint to restart on the data-plane
cluster instead.
Test plan
helm templatediff showsdefault
fullmode unchanged vs master).the original PR thread).
helm template --set installMode=controlPlaneOnlyrenders onlyovn-central + 3 Services +
ovn-ovsSA/RBAC (+ optional TLS Secret).helm template --set installMode=dataPlaneOnly --set externalOvnCentral.endpoint=<lb> --set externalOvnCentral.nbPort=31641 --set externalOvnCentral.sbPort=31642injects
OVN_NB_PORT=31641/OVN_SB_PORT=31642on both controllerand ovs-ovn DS.
helm template --set OVN_CENTRAL_MODE=single --set ovn-central.storage.existingClaim=fooskips in-chart PVC creation and
claimNamebecomesfoo.bash dist/images/restore-ovn-nb-db.sh single backup.dbon acontrolPlaneOnlycluster completes 0 with the ovs-ovn restartcleanly skipped.
Replaces
Supersedes #6782. That PR will be closed in favor of this one.
🤖 Generated with Claude Code