Skip to content

Commit 6d4b759

Browse files
rollandfrollandf
committed
feat: add perNodeExclusions support for IPPool and CIDRPool
Add support for per-node index-based IP exclusions in both IPPool and CIDRPool resources. This allows administrators to exclude specific IP address ranges based on their index position within each node's allocated IP block or prefix, rather than by absolute IP addresses. API Changes: - Add ExcludeIndexRange type with startIndex and endIndex fields - Add perNodeExclusions field to IPPoolSpec and CIDRPoolSpec - Add kubebuilder validation markers for CRD-level validation - startIndex and endIndex must be non-negative - endIndex must be >= startIndex - Add runtime validation functions: - validatePerNodeExclusions() for CIDRPool (validates against perNodeNetworkPrefix) - validatePerNodeExclusionsForBlockSize() for IPPool (validates against perNodeBlockSize) - Add comprehensive test coverage (24 test cases total) - 14 tests for CIDRPool (IPv4 and IPv6) - 10 tests for IPPool (IPv4 and IPv6) Controller Implementation: - IPPool controller: Add buildPerNodeExclusions() function - Converts index-based exclusions to IP addresses for node's IP block - Merges with pool-wide exclusions before passing to allocator - 10 unit tests covering all edge cases - CIDRPool controller: Add buildPerNodeExclusions() function - Converts index-based exclusions to IP addresses for node's prefix - Combines with filtered pool-wide exclusions - 7 unit tests covering all edge cases Implementation details: - Each node processes only its own allocation (per-node) - Index-based ranges are converted to IP-based ExclusionRange objects - Ranges are clamped to node's allocation boundaries - Ranges outside node's allocation are skipped - Reuses existing allocator exclusion mechanism (no allocator changes) - Memory efficient: large ranges create single ExclusionRange entries Documentation: - Update README.md with perNodeExclusions field descriptions - Update docs/static-ip.md with examples - Add detailed field explanations for both IPPool and CIDRPool Example usage: perNodeExclusions: - startIndex: 0 endIndex: 10 Node-A (192.168.0.1-192.168.0.100) excludes 192.168.0.1-192.168.0.11 Node-B (192.168.0.101-192.168.0.200) excludes 192.168.0.101-192.168.0.111 Signed-off-by: Fred Rolland <frolland@nvidia.com>
1 parent ce5b56b commit 6d4b759

19 files changed

+789
-6
lines changed

‎README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,9 @@ spec:
345345
exclusions: # optional
346346
- startIP: 192.168.0.10
347347
endIP: 192.168.0.20
348+
perNodeExclusions: # optional
349+
- startIndex: 0
350+
endIndex: 10
348351
nodeSelector:
349352
nodeSelectorTerms:
350353
- matchExpressions:
@@ -385,6 +388,11 @@ spec:
385388
* `startIP`: start IP of the exclude range (inclusive).
386389
* `endIP`: end IP of the exclude range (inclusive).
387390

391+
* `perNodeExclusions` (optional, list): contains reserved IP address indexes that should not be allocated by nv-ipam node component. The IP address indexes are relative to the per-node IP block allocated to each node, counting from the start of the allocated range. For example, if a node is allocated the IP block 192.168.0.1-192.168.0.100, then index 0 corresponds to 192.168.0.1, index 1 to 192.168.0.2, and so on. Note: For IPPool, indexes count from the range start; for CIDRPool, indexes count from the subnet start (network address).
392+
393+
* `startIndex`: start index of the exclude range (inclusive). Must be non-negative.
394+
* `endIndex`: end index of the exclude range (inclusive). Must be greater than or equal to startIndex and within the perNodeBlockSize range.
395+
388396
* `nodeSelector` (optional): A list of node selector terms. The terms are ORed. Each term can have a list of matchExpressions that are ANDed. Only the nodes that match the provided labels will get assigned IP Blocks for the defined pool.
389397
* `defaultGateway` (optional): Add the pool gateway as default gateway in the pod static routes.
390398
* `routes` (optional, list): contains CIDR to be added in the pod static routes via the pool gateway.
@@ -419,6 +427,9 @@ spec:
419427
exclusions: # optional
420428
- startIP: 192.168.0.10
421429
endIP: 192.168.0.20
430+
perNodeExclusions: # optional
431+
- startIndex: 0
432+
endIndex: 10
422433
staticAllocations:
423434
- nodeName: node-33
424435
prefix: 192.168.33.0/24
@@ -478,6 +489,11 @@ spec:
478489
* `startIP`: start IP of the exclude range (inclusive).
479490
* `endIP`: end IP of the exclude range (inclusive).
480491

492+
* `perNodeExclusions` (optional, list): contains reserved IP address indexes that should not be allocated by nv-ipam node component. The IP address indexes are relative to the per-node prefix allocated to each node, counting from the subnet start (network address). For example, if a node is allocated the prefix 192.168.0.0/24, then index 0 corresponds to 192.168.0.0 (network address), index 1 to 192.168.0.1 (which would be the gateway if gatewayIndex is 1), index 2 to 192.168.0.2, and so on. This indexing is consistent with gatewayIndex.
493+
494+
* `startIndex`: start index of the exclude range (inclusive). Must be non-negative.
495+
* `endIndex`: end index of the exclude range (inclusive). Must be greater than or equal to startIndex and within the subnet size defined by perNodeNetworkPrefix.
496+
481497
* `staticAllocations` (optional, list): static allocations for the pool.
482498

483499
* `nodeName` (optional): name of the node for static allocation, can be empty in case if the prefix should be preallocated without assigning it for a specific node.

‎api/v1alpha1/cidrpool_test.go‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,88 @@ var _ = Describe("CIDRPool", func() {
284284
{StartIP: "10.10.33.25", EndIP: "10.10.33.33"},
285285
}, false, ContainSubstring("spec.exclusions[0]")),
286286
)
287+
DescribeTable("PerNodeExclusions",
288+
func(perNodeExclusions []v1alpha1.ExcludeIndexRange, isValid bool, errMatcher ...gomegaTypes.GomegaMatcher) {
289+
cidrPool := v1alpha1.CIDRPool{
290+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
291+
Spec: v1alpha1.CIDRPoolSpec{
292+
CIDR: "192.168.0.0/16",
293+
PerNodeNetworkPrefix: 24,
294+
PerNodeExclusions: perNodeExclusions,
295+
},
296+
}
297+
validatePoolAndCheckErr(&cidrPool, isValid, errMatcher...)
298+
},
299+
Entry("valid - range", []v1alpha1.ExcludeIndexRange{
300+
{StartIndex: 10, EndIndex: 20},
301+
}, true),
302+
Entry("valid - single index", []v1alpha1.ExcludeIndexRange{
303+
{StartIndex: 5, EndIndex: 5},
304+
}, true),
305+
Entry("valid - index 0", []v1alpha1.ExcludeIndexRange{
306+
{StartIndex: 0, EndIndex: 10},
307+
}, true),
308+
Entry("valid - max index for /24 (255)", []v1alpha1.ExcludeIndexRange{
309+
{StartIndex: 250, EndIndex: 255},
310+
}, true),
311+
Entry("valid - multiple ranges", []v1alpha1.ExcludeIndexRange{
312+
{StartIndex: 0, EndIndex: 10},
313+
{StartIndex: 100, EndIndex: 110},
314+
}, true),
315+
Entry("negative startIndex", []v1alpha1.ExcludeIndexRange{
316+
{StartIndex: -5, EndIndex: 20},
317+
}, false, ContainSubstring("spec.perNodeExclusions[0].startIndex")),
318+
Entry("negative endIndex", []v1alpha1.ExcludeIndexRange{
319+
{StartIndex: 5, EndIndex: -1},
320+
}, false, ContainSubstring("spec.perNodeExclusions[0].endIndex")),
321+
Entry("startIndex greater than endIndex", []v1alpha1.ExcludeIndexRange{
322+
{StartIndex: 25, EndIndex: 24},
323+
}, false, ContainSubstring("spec.perNodeExclusions[0]")),
324+
Entry("startIndex outside subnet range", []v1alpha1.ExcludeIndexRange{
325+
{StartIndex: 256, EndIndex: 260},
326+
}, false, ContainSubstring("spec.perNodeExclusions[0].startIndex"), ContainSubstring("outside")),
327+
Entry("endIndex outside subnet range", []v1alpha1.ExcludeIndexRange{
328+
{StartIndex: 250, EndIndex: 300},
329+
}, false, ContainSubstring("spec.perNodeExclusions[0].endIndex"), ContainSubstring("outside")),
330+
Entry("multiple errors in one entry", []v1alpha1.ExcludeIndexRange{
331+
{StartIndex: -5, EndIndex: -1},
332+
}, false,
333+
ContainSubstring("spec.perNodeExclusions[0].startIndex"),
334+
ContainSubstring("spec.perNodeExclusions[0].endIndex")),
335+
Entry("multiple entries with errors", []v1alpha1.ExcludeIndexRange{
336+
{StartIndex: -5, EndIndex: 20},
337+
{StartIndex: 300, EndIndex: 400},
338+
}, false,
339+
ContainSubstring("spec.perNodeExclusions[0]"),
340+
ContainSubstring("spec.perNodeExclusions[1]")),
341+
)
342+
It("PerNodeExclusions - IPv6", func() {
343+
cidrPool := v1alpha1.CIDRPool{
344+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
345+
Spec: v1alpha1.CIDRPoolSpec{
346+
CIDR: "fdf8:6aef:d1fe::/48",
347+
PerNodeNetworkPrefix: 120, // 2^8 = 256 addresses
348+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
349+
{StartIndex: 0, EndIndex: 10},
350+
{StartIndex: 250, EndIndex: 255},
351+
},
352+
},
353+
}
354+
validatePoolAndCheckErr(&cidrPool, true)
355+
})
356+
It("PerNodeExclusions - IPv6 invalid", func() {
357+
cidrPool := v1alpha1.CIDRPool{
358+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
359+
Spec: v1alpha1.CIDRPoolSpec{
360+
CIDR: "fdf8:6aef:d1fe::/48",
361+
PerNodeNetworkPrefix: 120,
362+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
363+
{StartIndex: 0, EndIndex: 300}, // 300 > 255
364+
},
365+
},
366+
}
367+
validatePoolAndCheckErr(&cidrPool, false, ContainSubstring("spec.perNodeExclusions[0].endIndex"))
368+
})
287369
DescribeTable("StaticAllocations",
288370
func(staticAllocations []v1alpha1.CIDRPoolStaticAllocation, isValid bool, errMatcher ...gomegaTypes.GomegaMatcher) {
289371
cidrPool := v1alpha1.CIDRPool{

‎api/v1alpha1/cidrpool_type.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ type CIDRPoolSpec struct {
4444
PerNodeNetworkPrefix int32 `json:"perNodeNetworkPrefix"`
4545
// contains reserved IP addresses that should not be allocated by nv-ipam
4646
Exclusions []ExcludeRange `json:"exclusions,omitempty"`
47+
// contains reserved indexes of IPs that should not be allocated by nv-ipam
48+
PerNodeExclusions []ExcludeIndexRange `json:"perNodeExclusions,omitempty"`
4749
// static allocations for the pool
4850
StaticAllocations []CIDRPoolStaticAllocation `json:"staticAllocations,omitempty"`
4951
// selector for nodes, if empty match all nodes

‎api/v1alpha1/cidrpool_validate.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@ func (r *CIDRPool) validateCIDR() field.ErrorList {
8888
r.Spec.GatewayIndex, "gateway index is outside of the node prefix"))
8989
}
9090
errList = append(errList, validateExclusions(network, r.Spec.Exclusions, field.NewPath("spec"))...)
91+
errList = append(errList, validatePerNodeExclusions(
92+
network, r.Spec.PerNodeNetworkPrefix, r.Spec.PerNodeExclusions, field.NewPath("spec"))...)
9193
errList = append(errList, r.validateStaticAllocations(network)...)
9294
return errList
9395
}

‎api/v1alpha1/common_type.go‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,19 @@ type ExcludeRange struct {
2020
EndIP string `json:"endIP"`
2121
}
2222

23+
// ExcludeIndexRange contains range of indexes of IPs to exclude from allocation
24+
// startIndex and endIndex are part of the ExcludeIndexRange
25+
//
26+
// +kubebuilder:validation:XValidation:rule="self.endIndex >= self.startIndex",message="endIndex must be greater than or equal to startIndex"
27+
//
28+
//nolint:lll // kubebuilder annotation exceeds line length
29+
type ExcludeIndexRange struct {
30+
// +kubebuilder:validation:Minimum=0
31+
StartIndex int `json:"startIndex"`
32+
// +kubebuilder:validation:Minimum=0
33+
EndIndex int `json:"endIndex"`
34+
}
35+
2336
// Route contains static route parameters
2437
type Route struct {
2538
// The destination of the route, in CIDR notation

‎api/v1alpha1/ippool_test.go‎

Lines changed: 168 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ var _ = Describe("Validate", func() {
249249
Spec: v1alpha1.IPPoolSpec{
250250
Subnet: "2001:db8:3333:4444::0/64",
251251
PerNodeBlockSize: 128,
252-
DefaultGateway: true,
252+
DefaultGateway: true,
253253
Routes: []v1alpha1.Route{
254254
{
255255
Dst: "::/0",
@@ -268,7 +268,7 @@ var _ = Describe("Validate", func() {
268268
Spec: v1alpha1.IPPoolSpec{
269269
Subnet: "192.168.0.0/16",
270270
PerNodeBlockSize: 128,
271-
DefaultGateway: true,
271+
DefaultGateway: true,
272272
Routes: []v1alpha1.Route{
273273
{
274274
Dst: "0.0.0.0/0",
@@ -281,4 +281,170 @@ var _ = Describe("Validate", func() {
281281
ContainSubstring("spec.routes"),
282282
)
283283
})
284+
It("Valid - PerNodeExclusions", func() {
285+
ipPool := v1alpha1.IPPool{
286+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
287+
Spec: v1alpha1.IPPoolSpec{
288+
Subnet: "192.168.0.0/16",
289+
PerNodeBlockSize: 128,
290+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
291+
{StartIndex: 0, EndIndex: 10},
292+
{StartIndex: 100, EndIndex: 127},
293+
},
294+
},
295+
}
296+
Expect(ipPool.Validate()).To(BeEmpty())
297+
})
298+
It("Valid - PerNodeExclusions single index", func() {
299+
ipPool := v1alpha1.IPPool{
300+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
301+
Spec: v1alpha1.IPPoolSpec{
302+
Subnet: "192.168.0.0/16",
303+
PerNodeBlockSize: 128,
304+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
305+
{StartIndex: 5, EndIndex: 5},
306+
},
307+
},
308+
}
309+
Expect(ipPool.Validate()).To(BeEmpty())
310+
})
311+
It("Valid - PerNodeExclusions max index", func() {
312+
ipPool := v1alpha1.IPPool{
313+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
314+
Spec: v1alpha1.IPPoolSpec{
315+
Subnet: "192.168.0.0/16",
316+
PerNodeBlockSize: 128,
317+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
318+
{StartIndex: 120, EndIndex: 127}, // 127 is max for block size 128
319+
},
320+
},
321+
}
322+
Expect(ipPool.Validate()).To(BeEmpty())
323+
})
324+
It("Invalid - PerNodeExclusions negative startIndex", func() {
325+
ipPool := v1alpha1.IPPool{
326+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
327+
Spec: v1alpha1.IPPoolSpec{
328+
Subnet: "192.168.0.0/16",
329+
PerNodeBlockSize: 128,
330+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
331+
{StartIndex: -5, EndIndex: 20},
332+
},
333+
},
334+
}
335+
Expect(ipPool.Validate().ToAggregate().Error()).
336+
To(
337+
ContainSubstring("spec.perNodeExclusions[0].startIndex"),
338+
)
339+
})
340+
It("Invalid - PerNodeExclusions negative endIndex", func() {
341+
ipPool := v1alpha1.IPPool{
342+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
343+
Spec: v1alpha1.IPPoolSpec{
344+
Subnet: "192.168.0.0/16",
345+
PerNodeBlockSize: 128,
346+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
347+
{StartIndex: 5, EndIndex: -1},
348+
},
349+
},
350+
}
351+
Expect(ipPool.Validate().ToAggregate().Error()).
352+
To(
353+
ContainSubstring("spec.perNodeExclusions[0].endIndex"),
354+
)
355+
})
356+
It("Invalid - PerNodeExclusions startIndex greater than endIndex", func() {
357+
ipPool := v1alpha1.IPPool{
358+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
359+
Spec: v1alpha1.IPPoolSpec{
360+
Subnet: "192.168.0.0/16",
361+
PerNodeBlockSize: 128,
362+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
363+
{StartIndex: 25, EndIndex: 24},
364+
},
365+
},
366+
}
367+
Expect(ipPool.Validate().ToAggregate().Error()).
368+
To(
369+
ContainSubstring("spec.perNodeExclusions[0]"),
370+
)
371+
})
372+
It("Invalid - PerNodeExclusions startIndex outside block range", func() {
373+
ipPool := v1alpha1.IPPool{
374+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
375+
Spec: v1alpha1.IPPoolSpec{
376+
Subnet: "192.168.0.0/16",
377+
PerNodeBlockSize: 128,
378+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
379+
{StartIndex: 128, EndIndex: 130}, // max is 127 for block size 128
380+
},
381+
},
382+
}
383+
Expect(ipPool.Validate().ToAggregate().Error()).
384+
To(And(
385+
ContainSubstring("spec.perNodeExclusions[0].startIndex"),
386+
ContainSubstring("outside"),
387+
))
388+
})
389+
It("Invalid - PerNodeExclusions endIndex outside block range", func() {
390+
ipPool := v1alpha1.IPPool{
391+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
392+
Spec: v1alpha1.IPPoolSpec{
393+
Subnet: "192.168.0.0/16",
394+
PerNodeBlockSize: 128,
395+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
396+
{StartIndex: 120, EndIndex: 200},
397+
},
398+
},
399+
}
400+
Expect(ipPool.Validate().ToAggregate().Error()).
401+
To(And(
402+
ContainSubstring("spec.perNodeExclusions[0].endIndex"),
403+
ContainSubstring("outside"),
404+
))
405+
})
406+
It("Valid - PerNodeExclusions IPv6", func() {
407+
ipPool := v1alpha1.IPPool{
408+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
409+
Spec: v1alpha1.IPPoolSpec{
410+
Subnet: "2001:db8:3333:4444::0/64",
411+
PerNodeBlockSize: 256,
412+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
413+
{StartIndex: 0, EndIndex: 10},
414+
{StartIndex: 250, EndIndex: 255},
415+
},
416+
},
417+
}
418+
Expect(ipPool.Validate()).To(BeEmpty())
419+
})
420+
It("PerNodeExclusions - IPv6 large subnet", func() {
421+
cidrPool := v1alpha1.CIDRPool{
422+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
423+
Spec: v1alpha1.CIDRPoolSpec{
424+
CIDR: "fdf8:6aef:d1fe::/48",
425+
PerNodeNetworkPrefix: 64,
426+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
427+
{StartIndex: 0, EndIndex: 10},
428+
{StartIndex: 250, EndIndex: 255},
429+
},
430+
},
431+
}
432+
validatePoolAndCheckErr(&cidrPool, true)
433+
})
434+
It("Invalid - PerNodeExclusions IPv6 outside range", func() {
435+
ipPool := v1alpha1.IPPool{
436+
ObjectMeta: metav1.ObjectMeta{Name: "test"},
437+
Spec: v1alpha1.IPPoolSpec{
438+
Subnet: "2001:db8:3333:4444::0/64",
439+
PerNodeBlockSize: 256,
440+
PerNodeExclusions: []v1alpha1.ExcludeIndexRange{
441+
{StartIndex: 0, EndIndex: 300}, // max is 255 for block size 256
442+
},
443+
},
444+
}
445+
Expect(ipPool.Validate().ToAggregate().Error()).
446+
To(
447+
ContainSubstring("spec.perNodeExclusions[0].endIndex"),
448+
)
449+
})
284450
})

‎api/v1alpha1/ippool_type.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ type IPPoolSpec struct {
4141
PerNodeBlockSize int `json:"perNodeBlockSize"`
4242
// contains reserved IP addresses that should not be allocated by nv-ipam
4343
Exclusions []ExcludeRange `json:"exclusions,omitempty"`
44+
// contains reserved indexes of IPs that should not be allocated by nv-ipam
45+
PerNodeExclusions []ExcludeIndexRange `json:"perNodeExclusions,omitempty"`
4446
// gateway for the pool
4547
Gateway string `json:"gateway,omitempty"`
4648
// selector for nodes, if empty match all nodes

‎api/v1alpha1/ippool_validate.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,10 @@ func (r *IPPool) Validate() field.ErrorList {
5252
if network != nil {
5353
errList = append(errList, validateExclusions(network, r.Spec.Exclusions, field.NewPath("spec"))...)
5454
}
55+
if r.Spec.PerNodeBlockSize >= 1 {
56+
errList = append(errList, validatePerNodeExclusionsForBlockSize(
57+
r.Spec.PerNodeBlockSize, r.Spec.PerNodeExclusions, field.NewPath("spec"))...)
58+
}
5559
var parsedGW net.IP
5660
if r.Spec.Gateway != "" {
5761
parsedGW = net.ParseIP(r.Spec.Gateway)

0 commit comments

Comments
 (0)