Skip to content

Commit bb30324

Browse files
cleoz97Vishal Mohanoilbeater
authored
Tests: Added go unit tests and E2E tests for Kube-OVN Non-Primary CNI mode (#5732)
* Add unit tests for VPC NAT gateway and network attachment functionalities - Implemented tests for replacing endpoint addresses with secondary IPs in endpoint slices. - Created new test file for pod-related functionalities, including checks for VPC NAT gateway annotations and network attachments. - Added tests for retrieving subnet providers, ensuring correct behavior for valid and non-existent subnets. - Enhanced network attachment utility tests to cover various scenarios, including valid and invalid JSON inputs. Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> * Add e2e tests and configurations for non-primary CNI functionality - Implemented e2e tests for non-primary CNI, including VPC and logical network scenarios. - Created test configurations for VPC simple and NAT gateway setups. - Added YAML files for logical network and VPC resources to support dynamic KIND bridge network detection. - Enhanced test framework to handle network interface creation and configuration processing. Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> * Enhance E2E tests for non-primary CNI with dynamic versioning and resource cleanup - Introduced functions to dynamically determine Kube-OVN version and registry. - Added a cleanup function to remove finalizers from Kube-OVN resources. - Improved pod IP retrieval logic to handle both primary and non-primary CNI cases. - Refactored the processConfigWithKindBridge function to use dynamic image tags. - Enhanced test configurations to create a dedicated namespace and network attachment definitions for non-primary CNI testing. - Updated YAML configurations to reflect new resource definitions and annotations for test pods. - Improved error handling and resource readiness checks in tests. Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> * Clean up Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> * Fixed lint errors Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> * Add non-primary CNI E2E test workflow to build-x86-image.yaml Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> * Refactor non-primary CNI setup and enhance E2E tests with updated configurations Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> * reorder documents Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com> --------- Signed-off-by: Vishal Mohan <vishalmohan@microsoft.com> Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com> Co-authored-by: Vishal Mohan <vishalmohan@microsoft.com> Co-authored-by: Mengxin Liu <liumengxinfly@gmail.com>
1 parent 638d31d commit bb30324

File tree

17 files changed

+3340
-27
lines changed

17 files changed

+3340
-27
lines changed

‎.github/workflows/build-x86-image.yaml‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1564,6 +1564,162 @@ jobs:
15641564
if: ${{ success() || (failure() && (steps.install.conclusion == 'failure' || steps.e2e.conclusion == 'failure')) }}
15651565
run: make check-kube-ovn-pod-restarts
15661566

1567+
non-primary-cni-e2e:
1568+
name: Non-Primary CNI E2E
1569+
needs:
1570+
- build-kube-ovn
1571+
- build-e2e-binaries
1572+
runs-on: ubuntu-24.04
1573+
timeout-minutes: 30
1574+
strategy:
1575+
fail-fast: false
1576+
matrix:
1577+
ip-family:
1578+
- ipv4
1579+
steps:
1580+
- uses: jlumbroso/free-disk-space@v1.3.1
1581+
with:
1582+
android: true
1583+
dotnet: true
1584+
haskell: true
1585+
docker-images: false
1586+
large-packages: false
1587+
tool-cache: false
1588+
swap-storage: false
1589+
1590+
- uses: actions/checkout@v5
1591+
1592+
- name: Create the default branch directory
1593+
if: (github.base_ref || github.ref_name) != github.event.repository.default_branch
1594+
run: mkdir -p test/e2e/source
1595+
1596+
- name: Check out the default branch
1597+
if: (github.base_ref || github.ref_name) != github.event.repository.default_branch
1598+
uses: actions/checkout@v5
1599+
with:
1600+
ref: ${{ github.event.repository.default_branch }}
1601+
fetch-depth: 1
1602+
path: test/e2e/source
1603+
1604+
- name: Export E2E directory
1605+
run: |
1606+
if [ '${{ github.base_ref || github.ref_name }}' = '${{ github.event.repository.default_branch }}' ]; then
1607+
echo "E2E_DIR=." >> "$GITHUB_ENV"
1608+
else
1609+
echo "E2E_DIR=test/e2e/source" >> "$GITHUB_ENV"
1610+
fi
1611+
1612+
- name: Remove DNS search domain
1613+
run: |
1614+
sudo sed -i '/^search/d' /etc/resolv.conf
1615+
sudo systemctl restart docker
1616+
1617+
- uses: actions/setup-go@v6
1618+
id: setup-go
1619+
with:
1620+
go-version-file: ${{ env.E2E_DIR }}/go.mod
1621+
check-latest: true
1622+
cache: false
1623+
1624+
- name: Export Go full version
1625+
run: echo "GO_VERSION=${{ steps.setup-go.outputs.go-version }}" >> "$GITHUB_ENV"
1626+
1627+
- name: Go cache
1628+
uses: actions/cache/restore@v4
1629+
with:
1630+
path: |
1631+
~/.cache/go-build
1632+
~/go/pkg/mod
1633+
key: ${{ runner.os }}-e2e-go-${{ env.GO_VERSION }}-x86-${{ hashFiles(format('{0}/**/go.sum', env.E2E_DIR)) }}
1634+
restore-keys: ${{ runner.os }}-e2e-go-${{ env.GO_VERSION }}-x86-
1635+
1636+
- name: Install kind
1637+
uses: helm/kind-action@v1.12.0
1638+
with:
1639+
version: ${{ env.KIND_VERSION }}
1640+
install_only: true
1641+
1642+
- name: Install ginkgo
1643+
working-directory: ${{ env.E2E_DIR }}
1644+
run: go install -v -mod=mod github.com/onsi/ginkgo/v2/ginkgo
1645+
1646+
- name: Download image
1647+
uses: actions/download-artifact@v5
1648+
with:
1649+
name: kube-ovn
1650+
1651+
- name: Download vpc-nat-gateway image
1652+
uses: actions/download-artifact@v5
1653+
with:
1654+
name: vpc-nat-gateway
1655+
1656+
- name: Load image
1657+
run: |
1658+
docker load --input kube-ovn.tar
1659+
docker load --input vpc-nat-gateway.tar
1660+
1661+
- name: Create kind cluster and install CNIs
1662+
id: install
1663+
env:
1664+
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
1665+
run: |
1666+
pipx install jinjanator
1667+
make kind-ghcr-pull kind-install-multus-cilium-kubeovn-non-primary-${{ matrix.ip-family }}
1668+
1669+
- name: Run non-primary CNI e2e
1670+
id: e2e
1671+
working-directory: ${{ env.E2E_DIR }}
1672+
env:
1673+
E2E_BRANCH: ${{ github.base_ref || github.ref_name }}
1674+
E2E_IP_FAMILY: ${{ matrix.ip-family }}
1675+
E2E_NETWORK_MODE: overlay
1676+
KUBE_OVN_PRIMARY_CNI: "false"
1677+
run: make kube-ovn-non-primary-cni-e2e
1678+
1679+
- name: Collect k8s events
1680+
if: failure() && steps.e2e.conclusion == 'failure'
1681+
run: |
1682+
kubectl get events -A -o yaml > non-primary-cni-e2e-${{ matrix.ip-family }}-events.yaml
1683+
tar zcf non-primary-cni-e2e-${{ matrix.ip-family }}-events.tar.gz non-primary-cni-e2e-${{ matrix.ip-family }}-events.yaml
1684+
1685+
- name: Upload k8s events
1686+
uses: actions/upload-artifact@v4
1687+
if: failure() && steps.e2e.conclusion == 'failure'
1688+
with:
1689+
name: non-primary-cni-e2e-${{ matrix.ip-family }}-events
1690+
path: non-primary-cni-e2e-${{ matrix.ip-family }}-events.tar.gz
1691+
1692+
- name: Collect apiserver audit logs
1693+
if: failure() && steps.e2e.conclusion == 'failure'
1694+
run: |
1695+
docker cp kube-ovn-control-plane:/var/log/kubernetes/kube-apiserver-audit.log .
1696+
tar zcf non-primary-cni-e2e-${{ matrix.ip-family }}-audit-log.tar.gz kube-apiserver-audit.log
1697+
1698+
- name: Upload apiserver audit logs
1699+
uses: actions/upload-artifact@v4
1700+
if: failure() && steps.e2e.conclusion == 'failure'
1701+
with:
1702+
name: non-primary-cni-e2e-${{ matrix.ip-family }}-audit-log
1703+
path: non-primary-cni-e2e-${{ matrix.ip-family }}-audit-log.tar.gz
1704+
1705+
- name: kubectl ko log
1706+
if: failure() && steps.e2e.conclusion == 'failure'
1707+
working-directory: ${{ env.E2E_DIR }}
1708+
run: |
1709+
make kubectl-ko-log
1710+
mv kubectl-ko-log.tar.gz non-primary-cni-e2e-${{ matrix.ip-family }}-ko-log.tar.gz
1711+
1712+
- name: upload kubectl ko log
1713+
uses: actions/upload-artifact@v4
1714+
if: failure() && steps.e2e.conclusion == 'failure'
1715+
with:
1716+
name: non-primary-cni-e2e-${{ matrix.ip-family }}-ko-log
1717+
path: non-primary-cni-e2e-${{ matrix.ip-family }}-ko-log.tar.gz
1718+
1719+
- name: Check kube ovn pod restarts
1720+
if: ${{ success() || (failure() && (steps.install.conclusion == 'failure' || steps.e2e.conclusion == 'failure')) }}
1721+
run: make check-kube-ovn-pod-restarts
1722+
15671723
chart-test:
15681724
name: Chart Installation/Uninstallation Test
15691725
needs: build-kube-ovn
@@ -3752,6 +3908,7 @@ jobs:
37523908
- kube-ovn-ipsec-cert-mgr-e2e
37533909
- kube-ovn-underlay-metallb-e2e
37543910
- multus-conformance-e2e
3911+
- non-primary-cni-e2e
37553912
- vpc-egress-gateway-e2e
37563913
- ovn-vpc-nat-gw-conformance-e2e
37573914
- iptables-vpc-nat-gw-conformance-e2e

‎Makefile‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,9 @@ install-chart:
208208
--set func.ENABLE_OVN_IPSEC=$(or $(ENABLE_OVN_IPSEC),false) \
209209
--set func.ENABLE_TPROXY=$(or $(ENABLE_TPROXY),false) \
210210
--set func.ENABLE_IC=$(shell kubectl get node --show-labels | grep -qw "ovn.kubernetes.io/ic-gw" && echo true || echo false) \
211-
--set func.ENABLE_ANP=$(or $(ENABLE_ANP),false)
211+
--set func.ENABLE_ANP=$(or $(ENABLE_ANP),false) \
212+
--set cni_conf.NON_PRIMARY_CNI=$(or $(ENABLE_NON_PRIMARY_CNI),false) \
213+
--set cni_conf.CNI_CONFIG_PRIORITY=$(or $(CNI_CONFIG_PRIORITY),01)
212214

213215
.PHONY: upgrade-chart
214216
upgrade-chart:
@@ -237,17 +239,25 @@ upgrade-chart:
237239
--set func.ENABLE_OVN_IPSEC=$(or $(ENABLE_OVN_IPSEC),false) \
238240
--set func.ENABLE_TPROXY=$(or $(ENABLE_TPROXY),false) \
239241
--set func.ENABLE_IC=$(shell kubectl get node --show-labels | grep -qw "ovn.kubernetes.io/ic-gw" && echo true || echo false) \
240-
--set func.ENABLE_ANP=$(or $(ENABLE_ANP),false)
242+
--set func.ENABLE_ANP=$(or $(ENABLE_ANP),false) \
243+
--set cni_conf.NON_PRIMARY_CNI=$(or $(ENABLE_NON_PRIMARY_CNI),false) \
244+
--set cni_conf.CNI_CONFIG_PRIORITY=$(or $(CNI_CONFIG_PRIORITY),01)
241245
kubectl -n kube-system wait pod --for=condition=ready -l app=ovs --timeout=60s
242246

243247
.PHONY: install-chart-v2
244248
install-chart-v2:
245249
kubectl label node --overwrite -l node-role.kubernetes.io/control-plane kube-ovn/role=master
246-
helm install kubeovn ./charts/kube-ovn-v2 --wait
250+
helm install kubeovn ./charts/kube-ovn-v2 --wait \
251+
--set global.images.kubeovn.tag=$(VERSION) \
252+
--set cni.nonPrimaryCNI=$(or $(ENABLE_NON_PRIMARY_CNI),false) \
253+
--set cni.configPriority=$(or $(CNI_CONFIG_PRIORITY),01)
247254

248255
.PHONY: upgrade-chart-v2
249256
upgrade-chart-v2:
250-
helm upgrade kubeovn ./charts/kube-ovn-v2 --wait
257+
helm upgrade kubeovn ./charts/kube-ovn-v2 --wait \
258+
--set global.images.kubeovn.tag=$(VERSION) \
259+
--set cni.nonPrimaryCNI=$(or $(ENABLE_NON_PRIMARY_CNI),false) \
260+
--set cni.configPriority=$(or $(CNI_CONFIG_PRIORITY),01)
251261
kubectl -n kube-system wait pod --for=condition=ready -l app=ovs --timeout=60s
252262

253263
.PHONY: uninstall

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Kube-OVN, a [CNCF Sandbox Project](https://www.cncf.io/sandbox-projects/), integ
2020
- **Vlan/Underlay Support**: In addition to overlay network, Kube-OVN also supports underlay and vlan mode network for better performance and direct connectivity with physical network.
2121
- **Static IP Addresses for Workloads**: Allocate random or static IP addresses to workloads.
2222
- **Seamless VM LiveMigration**: Live migrate KubeVirt vm without network interruption.
23+
- **Non-Primary CNI Mode**: Kube-OVN can work as a secondary CNI alongside other primary CNIs (Cilium, Calico, etc.), providing additional network interfaces and advanced networking features via Network Attachment Definitions (NADs).
2324
- **Multi-Cluster Network**: Connect different Kubernetes/Openstack clusters into one L3 network.
2425
- **TroubleShooting Tools**: Handy tools to diagnose, trace, monitor and dump container network traffic to help troubleshoot complicate network issues.
2526
- **Prometheus & Grafana Integration**: Exposing network quality metrics like pod/node/service/dns connectivity/latency in Prometheus format.

‎docs/NON-PRIMARY-CNI-QUICKSTART.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Quick Start: Kube-OVN Non-Primary CNI
2+
3+
This guide provides a quick example of running Kube-OVN as a secondary CNI alongside Cilium.
4+
5+
## Prerequisites
6+
7+
- Kubernetes cluster with Cilium as primary CNI
8+
- Multus CNI installed
9+
- Helm 3.x
10+
11+
## Step 1: Install Kube-OVN in Non-Primary Mode
12+
13+
```bash
14+
helm install kube-ovn ./charts/kube-ovn-v2 \
15+
--namespace kube-system \
16+
--set cni.nonPrimaryCNI=true
17+
```
18+
19+
## Step 2: Create a VPC and Subnet
20+
21+
```yaml
22+
# vpc-secondary.yaml
23+
apiVersion: kubeovn.io/v1
24+
kind: Vpc
25+
metadata:
26+
name: vpc-secondary
27+
spec:
28+
namespaces:
29+
- default
30+
---
31+
apiVersion: kubeovn.io/v1
32+
kind: Subnet
33+
metadata:
34+
name: subnet-secondary
35+
spec:
36+
vpc: vpc-secondary
37+
cidr: "10.100.0.0/16"
38+
gateway: "10.100.0.1"
39+
provider: kube-ovn-secondary.default.ovn
40+
```
41+
42+
Apply the configuration:
43+
```bash
44+
kubectl apply -f vpc-secondary.yaml
45+
```
46+
47+
## Step 3: Create Network Attachment Definition
48+
49+
```yaml
50+
# nad-secondary.yaml
51+
apiVersion: k8s.cni.cncf.io/v1
52+
kind: NetworkAttachmentDefinition
53+
metadata:
54+
name: kube-ovn-secondary
55+
namespace: default
56+
spec:
57+
config: |
58+
{
59+
"cniVersion": "0.3.1",
60+
"type": "kube-ovn",
61+
"server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
62+
"provider": "kube-ovn-secondary.default.ovn"
63+
}
64+
```
65+
66+
Apply the NAD:
67+
```bash
68+
kubectl apply -f nad-secondary.yaml
69+
```
70+
71+
## Step 4: Deploy a Multi-Network Pod
72+
73+
```yaml
74+
# multi-network-pod.yaml
75+
apiVersion: v1
76+
kind: Pod
77+
metadata:
78+
name: multi-network-pod
79+
annotations:
80+
k8s.v1.cni.cncf.io/networks: default/kube-ovn-secondary
81+
spec:
82+
containers:
83+
- name: app
84+
image: nginx
85+
ports:
86+
- containerPort: 80
87+
```
88+
89+
Deploy the pod:
90+
```bash
91+
kubectl apply -f multi-network-pod.yaml
92+
```
93+
94+
## Step 5: Verify the Setup
95+
96+
Check the pod has multiple interfaces:
97+
```bash
98+
kubectl exec multi-network-pod -- ip addr show
99+
```
100+
101+
Expected output:
102+
```
103+
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
104+
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
105+
inet 127.0.0.1/8 scope host lo
106+
107+
2: eth0@if123: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
108+
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
109+
inet 10.244.0.5/24 brd 10.244.0.255 scope global eth0 # Cilium interface
110+
111+
3: net1@if124: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
112+
link/ether 00:00:00:ab:cd:ef brd ff:ff:ff:ff:ff:ff link-netnsid 0
113+
inet 10.100.0.2/16 brd 10.100.255.255 scope global net1 # Kube-OVN interface
114+
```
115+
116+
Check network status annotation:
117+
```bash
118+
kubectl get pod multi-network-pod -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/networks-status}' | jq .
119+
```
120+
121+
## Cleanup
122+
123+
```bash
124+
kubectl delete pod multi-network-pod
125+
kubectl delete nad kube-ovn-secondary
126+
kubectl delete -f vpc-secondary.yaml
127+
helm uninstall kube-ovn -n kube-system
128+
```
129+
130+
## Next Steps
131+
132+
- See [NON-PRIMARY-CNI.md](NON-PRIMARY-CNI.md) for detailed documentation
133+
- Explore VPC NAT Gateway configuration for external connectivity
134+
- Configure security groups and QoS policies for secondary networks

0 commit comments

Comments
 (0)