Skip to content

Commit 95444b3

Browse files
danmanordanmanor
committed
MGMT-21201: Enable dual-stack clusters
The lifecycle-agent is a core component responsible for managing Image-Based Upgrades (IBU) and Image-Based Install (IBI) operations in OpenShift Single Node OpenShift (SNO) clusters. It handles critical cluster lifecycle operations including: - **Network Configuration Management**: Configuring node IPs, machine networks, cluster networks, and service networks during cluster transitions - **Recertification (Recert)**: Re-signing certificates with updated cluster information (IPs, hostnames, etc.) when transforming seed images into target clusters - **Post-Pivot Operations**: Managing network setup, DNS configuration, and kubelet configuration after cluster pivot operations - **Seed Cluster Information**: Capturing and transforming network details from seed clusters for target cluster deployment Currently, the lifecycle-agent's network handling is designed around single-stack networking, where clusters operate on either IPv4 OR IPv6, but not both simultaneously. All network-related data structures use single string fields (`NodeIP`, `MachineNetwork`) and the logic assumes a single IP address per network interface. Key components involved: - `api/seedreconfig`: Defines the `SeedReconfiguration` API for cluster transformation parameters - `utils/client_helper.go`: Extracts cluster network information from Kubernetes API - `lca-cli/postpivot`: Handles post-upgrade network configuration and kubelet setup - `internal/recert`: Manages certificate re-signing with updated network information - `lca-cli/seedclusterinfo`: Captures seed cluster network details for replication **[MGMT-21201](https://issues.redhat.com//browse/MGMT-21201)**: The lifecycle-agent needs to support dual-stack networking configurations where OpenShift clusters operate with both IPv4 and IPv6 addresses simultaneously on the same interfaces. 1. **Single IP Assumption**: All network fields (`NodeIP`, `MachineNetwork`) are single strings, preventing multiple IP support 2. **Limited Network Discovery**: Cluster info extraction only captures the first internal IP address 3. **Inadequate Recert Logic**: Certificate re-signing only handles single IP changes 4. **Kubelet Configuration**: Node IP hint generation assumes single network per stack 5. **Missing Test Coverage**: No validation for dual-stack scenarios - Support IPv4 + IPv6 dual-stack clusters in IBU/IBI operations - Maintain 100% backward compatibility with existing single-stack configurations - Handle multiple machine networks per IP family - Update recert logic to process multiple IP addresses in certificate SANs - Ensure proper kubelet configuration for dual-stack node IPs - **Added `NodeIPs []string`** to `SeedReconfiguration` and `SeedClusterInfo` for multiple node IPs - **Added `MachineNetworks []string`** to support multiple machine network CIDRs - **Preserved legacy fields** (`NodeIP`, `MachineNetwork`) for backward compatibility with precedence rules - **Enhanced `GetClusterInfo()`** to discover all internal node IPs via `getNodeInternalIPs()` - **Added `getMachineNetworks()`** to extract all machine networks from install config - **Implemented backward compatibility** by populating legacy fields with first array element - **Updated `setNodeIpHint()`** to generate space-separated IP hints: `KUBELET_NODEIP_HINT=<ip1> <ip2>` - **Refactored `setNodeIPIfNotProvided()`** to parse kubelet config from `/etc/systemd/system/kubelet.service.d/20-nodenet.conf` - **Added `parseKubeletNodeIPs()`** function to extract both `KUBELET_NODE_IP` and `KUBELET_NODE_IPS` environment variables - **Enhanced file validation** to check for both existence AND valid content before triggering `nodeip-configuration` service - **Implemented `slices.Equal()`** comparison for clean IP change detection - **Updated config IP format** to comma-separated list: `config.IP = "ip1,ip2"` for multiple addresses - **Enhanced certificate SAN rules** to include all old and new IP addresses in replacement rules - ✅ Single-stack IPv4 and IPv6 configurations - ✅ Dual-stack (IPv4 + IPv6) with both primary orders - ✅ Multiple networks per IP family - ✅ Legacy → new field migration scenarios - ✅ Error handling and edge cases - ✅ Backward compatibility validation **100% backward compatibility maintained**: - Existing single-stack clusters continue to work without modification - Legacy `NodeIP` and `MachineNetwork` fields preserved and populated - New fields (`NodeIPs`, `MachineNetworks`) take precedence when specified - All existing APIs and behavior unchanged for single-stack scenarios - **All existing tests pass** - no regressions introduced - **59 new test cases** covering all dual-stack scenarios - **Production-ready validation** for IPv4, IPv6, and dual-stack configurations - **Edge case coverage** including empty configs, invalid data, and service interactions
1 parent 8698fd7 commit 95444b3

File tree

12 files changed

+1338
-139
lines changed

12 files changed

+1338
-139
lines changed

‎api/seedreconfig/seedreconfig.go‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,15 @@ type SeedReconfiguration struct {
4949
InfraID string `json:"infra_id,omitempty"`
5050

5151
// The desired IP address of the SNO node.
52+
// Use NodeIPs instead.
5253
NodeIP string `json:"node_ip,omitempty"`
5354

55+
// The desired IP addresses of the SNO node. One for single stack
56+
// clusters, and two for dual-stack clusters.
57+
// Use this field instead of NodeIP.
58+
// +optional
59+
NodeIPs []string `json:"node_ips,omitempty"`
60+
5461
// The container registry used to host the release image of the seed cluster.
5562
ReleaseRegistry string `json:"release_registry,omitempty"`
5663

@@ -102,8 +109,19 @@ type SeedReconfiguration struct {
102109
// MachineNetwork is the subnet provided by user for the ocp cluster.
103110
// This will be used to create the node network and choose ip address for the node.
104111
// Equivalent to install-config.yaml's machineNetwork.
112+
// Use MachineNetworks instead.
105113
MachineNetwork string `json:"machine_network,omitempty"`
106114

115+
// MachineNetworks is the list of subnets provided by user.
116+
// For single stack ocp clusters, this will be a single subnet.
117+
// For dual-stack ocp clusters, this will be a list of two subnets.
118+
// This will be used to create the node network and choose ip addresses for the node.
119+
// Equivalent to install-config.yaml's machineNetworks.
120+
// Use this field instead of MachineNetwork.
121+
// If both MachineNetwork and MachineNetworks are specified, MachineNetworks takes precedence.
122+
// +optional
123+
MachineNetworks []string `json:"machine_networks,omitempty"`
124+
107125
// Proxy is the proxy settings for the cluster. Equivalent to
108126
// install-config.yaml's proxy. This will replace the proxy settings of the
109127
// seed cluster. During IBI, the HTTP and HTTPS and NO proxy settings are

‎go.sum‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -459,7 +459,7 @@ k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f h1:GA7//TjRY9yWGy1poLzYYJ
459459
k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f/go.mod h1:R/HEjbvWI0qdfb8viZUeVZm0X6IZnxAydC7YU42CMw4=
460460
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 h1:M3sRQVHv7vB20Xc2ybTt7ODCeFj6JSWYFzOFnYeS6Ro=
461461
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
462-
open-cluster-management.io/api v1.0.0 h1:54QllH9DTudCk6VrGt0q8CDsE3MghqJeTaTN4RHZpE0=
462+
open-cluster-management.io/api v1.0.0 h1:TQ2Xq/bteFnsUukR4Ty4vmiAaCvenBL3hZXYFYz7q+U=
463463
open-cluster-management.io/api v1.0.0/go.mod h1:/OeqXycNBZQoe3WG6ghuWsMgsKGuMZrK8ZpsU6gWL0Y=
464464
open-cluster-management.io/config-policy-controller v0.16.0 h1:hIrgTvJXYRuYZ+Gph4EPq+8heQKczWlCSSctGQVi6OI=
465465
open-cluster-management.io/config-policy-controller v0.16.0/go.mod h1:NEl+/esHnMIbRRwndgBqzILqL+nro2mbrcDowZHSMZk=

‎internal/clusterconfig/clusterconfig.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -309,6 +309,7 @@ func SeedReconfigurationFromClusterInfo(
309309
ClusterID: clusterInfo.ClusterID,
310310
InfraID: infraID,
311311
NodeIP: clusterInfo.NodeIP,
312+
NodeIPs: clusterInfo.NodeIPs,
312313
ReleaseRegistry: clusterInfo.ReleaseRegistry,
313314
Hostname: clusterInfo.Hostname,
314315
KubeconfigCryptoRetention: *kubeconfigCryptoRetention,
@@ -320,6 +321,7 @@ func SeedReconfigurationFromClusterInfo(
320321
StatusProxy: statusProxy,
321322
InstallConfig: installConfig,
322323
MachineNetwork: clusterInfo.MachineNetwork,
324+
MachineNetworks: clusterInfo.MachineNetworks,
323325
ChronyConfig: chronyConfig,
324326
AdditionalTrustBundle: seedreconfig.AdditionalTrustBundle{
325327
UserCaBundle: additionalTrustBundle.UserCaBundle,

‎internal/clusterconfig/clusterconfig_dualstack_test.go‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
package clusterconfig
2+
3+
import (
4+
"strings"
5+
"testing"
6+
7+
"github.com/openshift-kni/lifecycle-agent/lca-cli/seedclusterinfo"
8+
"github.com/openshift-kni/lifecycle-agent/utils"
9+
"github.com/stretchr/testify/assert"
10+
)
11+
12+
func TestSeedReconfigurationDifferentStack(t *testing.T) {
13+
testcases := []struct {
14+
name string
15+
clusterInfo *utils.ClusterInfo
16+
expectedNodeIP string
17+
expectedNodeIPs []string
18+
expectedMachineNetworks []string
19+
}{
20+
{
21+
name: "Single-stack IPv4",
22+
clusterInfo: &utils.ClusterInfo{
23+
OCPVersion: "4.15.0",
24+
BaseDomain: "example.com",
25+
ClusterName: "test-cluster",
26+
NodeIP: "192.168.1.10",
27+
NodeIPs: []string{"192.168.1.10"},
28+
MachineNetwork: "192.168.1.0/24",
29+
MachineNetworks: []string{"192.168.1.0/24"},
30+
ClusterNetworks: []string{"10.244.0.0/16"},
31+
ServiceNetworks: []string{"10.96.0.0/16"},
32+
Hostname: "test-node",
33+
},
34+
expectedNodeIP: "192.168.1.10",
35+
expectedNodeIPs: []string{"192.168.1.10"},
36+
expectedMachineNetworks: []string{"192.168.1.0/24"},
37+
},
38+
{
39+
name: "Single-stack IPv6",
40+
clusterInfo: &utils.ClusterInfo{
41+
OCPVersion: "4.15.0",
42+
BaseDomain: "example.com",
43+
ClusterName: "test-cluster",
44+
NodeIP: "2001:db8::10",
45+
NodeIPs: []string{"2001:db8::10"},
46+
MachineNetwork: "2001:db8::/64",
47+
MachineNetworks: []string{"2001:db8::/64"},
48+
ClusterNetworks: []string{"2001:db8:1::/64"},
49+
ServiceNetworks: []string{"2001:db8:2::/64"},
50+
Hostname: "test-node",
51+
},
52+
expectedNodeIP: "2001:db8::10",
53+
expectedNodeIPs: []string{"2001:db8::10"},
54+
expectedMachineNetworks: []string{"2001:db8::/64"},
55+
},
56+
{
57+
name: "Dual-stack IPv4 primary",
58+
clusterInfo: &utils.ClusterInfo{
59+
OCPVersion: "4.15.0",
60+
BaseDomain: "example.com",
61+
ClusterName: "test-cluster",
62+
NodeIP: "192.168.1.10",
63+
NodeIPs: []string{"192.168.1.10", "2001:db8::10"},
64+
MachineNetwork: "192.168.1.0/24",
65+
MachineNetworks: []string{"192.168.1.0/24", "2001:db8::/64"},
66+
ClusterNetworks: []string{"10.244.0.0/16", "2001:db8:1::/64"},
67+
ServiceNetworks: []string{"10.96.0.0/16", "2001:db8:2::/64"},
68+
Hostname: "test-node",
69+
},
70+
expectedNodeIP: "192.168.1.10",
71+
expectedNodeIPs: []string{"192.168.1.10", "2001:db8::10"},
72+
expectedMachineNetworks: []string{"192.168.1.0/24", "2001:db8::/64"},
73+
},
74+
{
75+
name: "Dual-stack IPv6 primary",
76+
clusterInfo: &utils.ClusterInfo{
77+
OCPVersion: "4.15.0",
78+
BaseDomain: "example.com",
79+
ClusterName: "test-cluster",
80+
NodeIP: "2001:db8::10",
81+
NodeIPs: []string{"2001:db8::10", "192.168.1.10"},
82+
MachineNetwork: "2001:db8::/64",
83+
MachineNetworks: []string{"2001:db8::/64", "192.168.1.0/24"},
84+
ClusterNetworks: []string{"2001:db8:1::/64", "10.244.0.0/16"},
85+
ServiceNetworks: []string{"2001:db8:2::/64", "10.96.0.0/16"},
86+
Hostname: "test-node",
87+
},
88+
expectedNodeIP: "2001:db8::10",
89+
expectedNodeIPs: []string{"2001:db8::10", "192.168.1.10"},
90+
expectedMachineNetworks: []string{"2001:db8::/64", "192.168.1.0/24"},
91+
},
92+
}
93+
94+
for _, tc := range testcases {
95+
t.Run(tc.name, func(t *testing.T) {
96+
// Create SeedClusterInfo from ClusterInfo
97+
seedClusterInfo := seedclusterinfo.NewFromClusterInfo(
98+
tc.clusterInfo,
99+
"quay.io/example/seed:latest",
100+
false, // hasProxy
101+
false, // hasFIPS
102+
nil, // additionalTrustBundle
103+
"", // containerStorageMountpointTarget
104+
"", // ingressCertificateCN
105+
)
106+
107+
// The SeedReconfiguration would be created from this SeedClusterInfo
108+
// For testing purposes, we verify that the SeedClusterInfo contains the expected data
109+
assert.Equal(t, tc.expectedNodeIP, seedClusterInfo.NodeIP)
110+
assert.Equal(t, tc.expectedNodeIPs, seedClusterInfo.NodeIPs)
111+
assert.Equal(t, tc.expectedMachineNetworks, seedClusterInfo.MachineNetworks)
112+
113+
// Verify that backward compatibility is maintained
114+
if len(tc.expectedNodeIPs) > 0 {
115+
assert.Equal(t, tc.expectedNodeIPs[0], seedClusterInfo.NodeIP, "First NodeIP should match legacy NodeIP field")
116+
}
117+
if len(tc.expectedMachineNetworks) > 0 {
118+
// Note: SeedClusterInfo doesn't have a single MachineNetwork field, only MachineNetworks slice
119+
assert.Contains(t, seedClusterInfo.MachineNetworks, tc.expectedMachineNetworks[0], "First machine network should be in the list")
120+
}
121+
})
122+
}
123+
}
124+
125+
func TestIPStackTypeDetermination(t *testing.T) {
126+
testcases := []struct {
127+
name string
128+
nodeIPs []string
129+
description string
130+
stackType string
131+
}{
132+
{
133+
name: "Single-stack IPv4",
134+
nodeIPs: []string{"192.168.1.10"},
135+
description: "Only IPv4 address",
136+
stackType: "IPv4",
137+
},
138+
{
139+
name: "Single-stack IPv6",
140+
nodeIPs: []string{"2001:db8::10"},
141+
description: "Only IPv6 address",
142+
stackType: "IPv6",
143+
},
144+
{
145+
name: "Dual-stack IPv4 primary",
146+
nodeIPs: []string{"192.168.1.10", "2001:db8::10"},
147+
description: "IPv4 and IPv6, IPv4 first",
148+
stackType: "dual-stack",
149+
},
150+
{
151+
name: "Dual-stack IPv6 primary",
152+
nodeIPs: []string{"2001:db8::10", "192.168.1.10"},
153+
description: "IPv4 and IPv6, IPv6 first",
154+
stackType: "dual-stack",
155+
},
156+
{
157+
name: "Multiple IPv4 addresses",
158+
nodeIPs: []string{"192.168.1.10", "10.0.0.5", "172.16.1.10"},
159+
description: "Multiple IPv4 addresses",
160+
stackType: "IPv4",
161+
},
162+
{
163+
name: "Multiple IPv6 addresses",
164+
nodeIPs: []string{"2001:db8::10", "2001:db9::20"},
165+
description: "Multiple IPv6 addresses",
166+
stackType: "IPv6",
167+
},
168+
{
169+
name: "Complex dual-stack",
170+
nodeIPs: []string{"192.168.1.10", "2001:db8::10", "10.0.0.5", "2001:db9::20"},
171+
description: "Multiple IPv4 and IPv6 addresses",
172+
stackType: "dual-stack",
173+
},
174+
}
175+
176+
for _, tc := range testcases {
177+
t.Run(tc.name, func(t *testing.T) {
178+
// Helper function to determine IP stack type
179+
stackType := determineIPStackType(tc.nodeIPs)
180+
assert.Equal(t, tc.stackType, stackType, tc.description)
181+
})
182+
}
183+
}
184+
185+
// Helper function to determine IP stack type for testing
186+
func determineIPStackType(ips []string) string {
187+
hasIPv4 := false
188+
hasIPv6 := false
189+
190+
for _, ip := range ips {
191+
if strings.Contains(ip, ":") {
192+
hasIPv6 = true
193+
} else {
194+
hasIPv4 = true
195+
}
196+
}
197+
198+
if hasIPv4 && hasIPv6 {
199+
return "dual-stack"
200+
} else if hasIPv6 {
201+
return "IPv6"
202+
} else {
203+
return "IPv4"
204+
}
205+
}

‎internal/recert/recert.go‎

Lines changed: 38 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import (
55
"fmt"
66
"os"
77
"path/filepath"
8+
"slices"
89
"strings"
910

1011
"github.com/openshift-kni/lifecycle-agent/api/seedreconfig"
@@ -14,6 +15,28 @@ import (
1415
"golang.org/x/crypto/bcrypt"
1516
)
1617

18+
// getAllNodeIPsFromSeedReconfig returns all node IPs with backward compatibility
19+
func getAllNodeIPsFromSeedReconfig(seedReconfig *seedreconfig.SeedReconfiguration) []string {
20+
if len(seedReconfig.NodeIPs) > 0 {
21+
return seedReconfig.NodeIPs
22+
}
23+
if seedReconfig.NodeIP != "" {
24+
return []string{seedReconfig.NodeIP}
25+
}
26+
return []string{}
27+
}
28+
29+
// getAllNodeIPsFromSeedClusterInfo returns all node IPs from seed cluster info
30+
func getAllNodeIPsFromSeedClusterInfo(seedClusterInfo *seedclusterinfo.SeedClusterInfo) []string {
31+
if len(seedClusterInfo.NodeIPs) > 0 {
32+
return seedClusterInfo.NodeIPs
33+
}
34+
if seedClusterInfo.NodeIP != "" {
35+
return []string{seedClusterInfo.NodeIP}
36+
}
37+
return []string{}
38+
}
39+
1740
const (
1841
RecertConfigFile = "recert_config.json"
1942
SummaryFile = "/var/tmp/recert-summary.yaml"
@@ -121,8 +144,12 @@ func CreateRecertConfigFile(seedReconfig *seedreconfig.SeedReconfiguration, seed
121144
config.Hostname = seedReconfig.Hostname
122145
}
123146

124-
if seedReconfig.NodeIP != seedClusterInfo.NodeIP {
125-
config.IP = seedReconfig.NodeIP
147+
allSeedNodeIPs := getAllNodeIPsFromSeedClusterInfo(seedClusterInfo)
148+
allNodeIPs := getAllNodeIPsFromSeedReconfig(seedReconfig)
149+
150+
ipsChanged := !slices.Equal(allSeedNodeIPs, allNodeIPs)
151+
if ipsChanged && len(allNodeIPs) > 0 {
152+
config.IP = strings.Join(allNodeIPs, ",")
126153
}
127154

128155
config.Proxy = FormatRecertProxyFromSeedReconfigProxy(seedReconfig.Proxy, seedReconfig.StatusProxy)
@@ -168,12 +195,19 @@ func CreateRecertConfigFile(seedReconfig *seedreconfig.SeedReconfiguration, seed
168195
config.CNSanReplaceRules = []string{
169196
fmt.Sprintf("system:node:%s,system:node:%s", seedClusterInfo.SNOHostname, seedReconfig.Hostname),
170197
fmt.Sprintf("%s,%s", seedClusterInfo.SNOHostname, seedReconfig.Hostname),
171-
fmt.Sprintf("%s,%s", seedClusterInfo.NodeIP, seedReconfig.NodeIP),
172198
fmt.Sprintf("api.%s,api.%s", seedFullDomain, clusterFullDomain),
173199
fmt.Sprintf("api-int.%s,api-int.%s", seedFullDomain, clusterFullDomain),
174200
fmt.Sprintf("*.apps.%s,*.apps.%s", seedFullDomain, clusterFullDomain),
175201
}
176-
// check if there is an ingress CN provided for backwards compatibility
202+
203+
if len(allSeedNodeIPs) == len(allNodeIPs) {
204+
for i := range allSeedNodeIPs {
205+
config.CNSanReplaceRules = append(
206+
config.CNSanReplaceRules, fmt.Sprintf("%s,%s", allSeedNodeIPs[i], allNodeIPs[i]),
207+
)
208+
}
209+
}
210+
177211
if seedReconfig.KubeconfigCryptoRetention.IngresssCrypto.IngressCertificateCN != "" {
178212
config.CNSanReplaceRules = append(config.CNSanReplaceRules,
179213
fmt.Sprintf("%s,%s", seedClusterInfo.IngressCertificateCN, seedReconfig.KubeconfigCryptoRetention.IngresssCrypto.IngressCertificateCN))

0 commit comments

Comments
 (0)