Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 24 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ The driver features an advanced resource filtering system that enables administr
## Features

- **Dynamic Resource Allocation**: Leverages Kubernetes DRA framework for SR-IOV VF management
- **Advanced Resource Filtering**: Fine-grained filtering of Virtual Functions based on hardware attributes
- **Custom Resource Definitions**: SriovResourceFilter CRD for configuring device filtering policies
- **Controller-based Management**: Kubernetes controller pattern for resource filter lifecycle management
- **Opt-In Device Advertisement**: Devices are only advertised when explicitly defined in a policy
- **Custom Resource Definitions**: SriovResourcePolicy CRD for configuring device advertisement policies
- **Controller-based Management**: Kubernetes controller pattern for resource policy lifecycle management
- **Multiple Resource Types**: Support for exposing different VF pools as distinct resource types
- **Node-targeted Filtering**: Per-node resource filtering with node selector support
- **Node-targeted Policies**: Per-node resource policies with node selector support
- **CDI Integration**: Uses Container Device Interface for device injection into containers
- **NRI Integration**: Node Resource Interface support for advanced container runtime interaction
- **Kubernetes Native**: Integrates seamlessly with standard Kubernetes resource request/limit model
Expand Down Expand Up @@ -79,7 +79,7 @@ The Helm chart supports various configuration options through `values.yaml`:
- **Image Configuration**: Customize image repository, tag, and pull policy
- **Resource Limits**: Set resource requests and limits for driver components
- **Node Selection**: Configure node selectors and tolerations
- **Namespace Configuration**: Configure the namespace where SriovResourceFilter resources are watched
- **Namespace Configuration**: Configure the namespace where SriovResourcePolicy resources are watched
- **Default Interface Prefix**: Set the default interface prefix for virtual functions
- **CDI Root**: Configure the directory for CDI file generation
- **Logging**: Adjust log verbosity and format
Expand Down Expand Up @@ -137,17 +137,19 @@ spec:

## Resource Filtering System

The DRA driver includes an advanced resource filtering system that allows administrators to define fine-grained policies for how SR-IOV Virtual Functions are exposed and allocated. This system uses Custom Resource Definitions (CRDs) and a Kubernetes controller to manage device filtering based on hardware characteristics.
The DRA driver uses an opt-in model where administrators explicitly define which SR-IOV Virtual Functions should be advertised as Kubernetes resources. This system uses Custom Resource Definitions (CRDs) and a Kubernetes controller to manage device advertisement policies based on hardware characteristics.

### SriovResourceFilter CRD
**Important**: Without a matching `SriovResourcePolicy`, no devices will be advertised.

The `SriovResourceFilter` custom resource allows you to define filtering policies for SR-IOV devices:
### SriovResourcePolicy CRD

The `SriovResourcePolicy` custom resource defines which SR-IOV devices should be advertised as allocatable resources:

```yaml
apiVersion: sriovnetwork.k8snetworkplumbingwg.io/v1alpha1
kind: SriovResourceFilter
kind: SriovResourcePolicy
metadata:
name: example-filter
name: example-policy
namespace: dra-sriov-driver
spec:
nodeSelector:
Expand Down Expand Up @@ -207,9 +209,9 @@ spec:
pfNames: ["eth1"]
```

### Using Filtered Resources
### Using Policy-Defined Resources

Once a `SriovResourceFilter` is applied, pods can request specific resource types using CEL expressions:
Once a `SriovResourcePolicy` is applied, devices matching the policy are advertised and pods can request specific resource types using CEL expressions:

```yaml
apiVersion: resource.k8s.io/v1
Expand Down Expand Up @@ -302,11 +304,11 @@ Demonstrates requesting multiple Virtual Functions in a single resource claim:
- VfConfig applies to all allocated VFs in the claim
- Automatic interface naming (typically net1, net2, etc.)

#### Resource Filtering (`demo/resource-filtering/`)
Shows how to use SriovResourceFilter for advanced device management:
- Filter VFs based on vendor ID, Physical Function names, and hardware attributes
#### Resource Policies (`demo/resource-policies/`)
Shows how to use SriovResourcePolicy for controlling device advertisement:
- Advertise VFs based on vendor ID, Physical Function names, and hardware attributes
- Multiple resource configurations for different network interfaces
- Node-targeted filtering with selector support
- Node-targeted policies with selector support

#### VFIO Driver Configuration (`demo/vfio-driver/`)
Illustrates VFIO-PCI driver configuration for userspace applications:
Expand All @@ -326,10 +328,10 @@ Illustrates VFIO-PCI driver configuration for userspace applications:
│ └── dra-driver-sriov/ # Main driver executable
├── pkg/
│ ├── driver/ # Core driver implementation
│ ├── controller/ # Kubernetes controller for resource filtering
│ ├── controller/ # Kubernetes controller for resource policies
│ ├── devicestate/ # Device state management and discovery
│ ├── api/ # API definitions
│ │ ├── sriovdra/v1alpha1/ # SriovResourceFilter CRD definitions
│ │ ├── sriovdra/v1alpha1/ # SriovResourcePolicy CRD definitions
│ │ └── virtualfunction/v1alpha1/ # Virtual Function API types
│ ├── cdi/ # CDI integration
│ ├── cni/ # CNI plugin integration
Expand All @@ -345,7 +347,7 @@ Illustrates VFIO-PCI driver configuration for userspace applications:
├── demo/ # Example workload configurations
│ ├── single-vf-claim/ # Single VF allocation example
│ ├── multiple-vf-claim/ # Multiple VF allocation example
│ ├── resource-filtering/ # Resource filtering configuration example
│ ├── resource-policies/ # Resource policy configuration example
│ └── vfio-driver/ # VFIO-PCI driver configuration example
├── hack/ # Build and development scripts
├── test/ # Test suites
Expand All @@ -355,9 +357,9 @@ Illustrates VFIO-PCI driver configuration for userspace applications:
### Key Components

- **Driver**: Main gRPC service implementing DRA kubelet plugin interface
- **Resource Filter Controller**: Kubernetes controller managing SriovResourceFilter lifecycle and device filtering
- **Device State Manager**: Tracks available and allocated SR-IOV virtual functions with filtering support
- **SriovResourceFilter CRD**: Custom resource for defining device filtering policies
- **Resource Policy Controller**: Kubernetes controller managing SriovResourcePolicy lifecycle and device advertisement
- **Device State Manager**: Tracks available and allocated SR-IOV virtual functions
- **SriovResourcePolicy CRD**: Custom resource for defining device advertisement policies (opt-in model)
- **CDI Generator**: Creates Container Device Interface specifications for VFs
- **NRI Plugin**: Node Resource Interface integration for container runtime interaction
- **Pod Manager**: Manages pod lifecycle and resource allocation
Expand Down
14 changes: 7 additions & 7 deletions cmd/dra-driver-sriov/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ func newApp() *cli.App {
},
&cli.StringFlag{
Name: "namespace",
Usage: "Namespace where the driver should watch for SriovResourceFilter resources.",
Usage: "Namespace where the driver should watch for SriovResourcePolicy resources.",
Value: "dra-sriov-driver",
Destination: &flagsOptions.Namespace,
EnvVars: []string{"NAMESPACE"},
Expand Down Expand Up @@ -188,11 +188,11 @@ func RunPlugin(ctx context.Context, config *types.Config) error {

logger.Info("Configuring controller manager", "namespace", config.Flags.Namespace)

// Configure cache to only watch resources in the specified namespace for SriovResourceFilter
// Configure cache to only watch resources in the specified namespace for SriovResourcePolicy
// while allowing cluster-wide access for other resources like Nodes
cacheOpts := cache.Options{
ByObject: map[client.Object]cache.ByObject{
&sriovdrav1alpha1.SriovResourceFilter{}: {
&sriovdrav1alpha1.SriovResourcePolicy{}: {
Namespaces: map[string]cache.Config{
config.Flags.Namespace: {},
},
Expand All @@ -209,10 +209,10 @@ func RunPlugin(ctx context.Context, config *types.Config) error {
return fmt.Errorf("failed to create controller manager: %w", err)
}

// create and setup resource filter controller
resourceFilterController := controller.NewSriovResourceFilterReconciler(config.K8sClient.Client, config.Flags.NodeName, config.Flags.Namespace, deviceStateManager)
if err := resourceFilterController.SetupWithManager(mgr); err != nil {
return fmt.Errorf("failed to setup resource filter controller: %w", err)
// create and setup resource policy controller
resourcePolicyController := controller.NewSriovResourcePolicyReconciler(config.K8sClient.Client, config.Flags.NodeName, config.Flags.Namespace, deviceStateManager)
if err := resourcePolicyController.SetupWithManager(mgr); err != nil {
return fmt.Errorf("failed to setup resource policy controller: %w", err)
}

// start controller manager
Expand Down
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# Resource Filtering Demo
# Resource Policy Demo

This demo showcases how to use `SriovResourceFilter` to filter and manage SR-IOV Virtual Functions based on various hardware and configuration criteria.
This demo showcases how to use `SriovResourcePolicy` to control which SR-IOV Virtual Functions are advertised as Kubernetes resources based on various hardware and configuration criteria.

## Overview

This scenario demonstrates:
- Creating resource filters based on vendor IDs, Physical Function names, and other hardware attributes
- Creating resource policies based on vendor IDs, Physical Function names, and other hardware attributes
- Setting up multiple resource configurations for different network interfaces
- Deploying a pod that uses filtered SR-IOV resources with specific network requirements
- Deploying a pod that uses policy-filtered SR-IOV resources with specific network requirements

## Components

### 1. SriovResourceFilter
The `SriovResourceFilter` resource defines how to filter available SR-IOV devices:
### 1. SriovResourcePolicy
The `SriovResourcePolicy` resource defines which SR-IOV devices should be advertised as allocatable resources:
- **nodeSelector**: Targets specific nodes (`dra-ctlplane-0.dra.lab` in this example)
- **configs**: Defines multiple resource configurations:
- `eth0_resource`: Filters devices connected to eth0 Physical Function
Expand All @@ -39,17 +39,20 @@ The `SriovResourceFilter` resource defines how to filter available SR-IOV device

## Usage

1. Apply the resource filter to make filtered resources available:
1. Apply the resource policy to advertise SR-IOV resources:
```bash
kubectl apply -f resource-filter.yaml
kubectl apply -f resource-policy.yaml
```

2. The DRA driver will discover and filter SR-IOV devices based on the criteria
3. Pods can then claim resources using the filtered resource names
2. The DRA driver will discover SR-IOV devices and advertise only those matching the policy criteria
3. Pods can then claim resources using the advertised resource names
4. The pod will be scheduled on nodes where matching resources are available

**Note**: Without a matching `SriovResourcePolicy`, no devices will be advertised (opt-in model).

## Key Features

- **Opt-In Model**: Devices are only advertised when explicitly defined in a policy
- **Granular Filtering**: Filter by vendor, device ID, PCI address, PF name, NUMA node, or driver
- **Multi-Resource Support**: Configure multiple resource types on the same node
- **CEL Integration**: Use Common Expression Language for advanced resource selection
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apiVersion: sriovnetwork.k8snetworkplumbingwg.io/v1alpha1
kind: SriovResourceFilter
kind: SriovResourcePolicy
metadata:
name: example-resource-filter
name: example-resource-policy
namespace: dra-sriov-driver
spec:
# NodeSelector to match specific nodes
Expand Down
4 changes: 2 additions & 2 deletions deployments/helm/dra-driver-sriov/templates/clusterrole.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,5 @@ rules:
resources: ["network-attachment-definitions"]
verbs: ["get"]
- apiGroups: ["sriovnetwork.k8snetworkplumbingwg.io"]
resources: ["sriovresourcefilters"]
verbs: ["get", "list", "watch"] # SriovResourceFilter resources
resources: ["sriovresourcepolicies"]
verbs: ["get", "list", "watch"] # SriovResourcePolicy resources
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,21 @@ kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.20.0
name: sriovresourcefilters.sriovnetwork.k8snetworkplumbingwg.io
name: sriovresourcepolicies.sriovnetwork.k8snetworkplumbingwg.io
spec:
group: sriovnetwork.k8snetworkplumbingwg.io
names:
kind: SriovResourceFilter
listKind: SriovResourceFilterList
plural: sriovresourcefilters
singular: sriovresourcefilter
kind: SriovResourcePolicy
listKind: SriovResourcePolicyList
plural: sriovresourcepolicies
singular: sriovresourcepolicy
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
description: SriovResourceFilter is a filter for SR-IOV resources
description: SriovResourcePolicy defines a policy for advertising SR-IOV devices
as Kubernetes resources
properties:
apiVersion:
description: |-
Expand All @@ -37,7 +38,7 @@ spec:
metadata:
type: object
spec:
description: SriovResourceFilterSpec is the spec for a SriovResourceFilter
description: SriovResourcePolicySpec is the spec for a SriovResourcePolicy
properties:
configs:
items:
Expand Down
18 changes: 9 additions & 9 deletions pkg/api/sriovdra/v1alpha1/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,21 +22,21 @@ import (

//nolint:gochecknoinits // Required for Kubernetes scheme registration
func init() {
SchemeBuilder.Register(&SriovResourceFilter{}, &SriovResourceFilterList{})
SchemeBuilder.Register(&SriovResourcePolicy{}, &SriovResourcePolicyList{})
}

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// SriovResourceFilter is a filter for SR-IOV resources
type SriovResourceFilter struct {
// SriovResourcePolicy defines a policy for advertising SR-IOV devices as Kubernetes resources
type SriovResourcePolicy struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec SriovResourceFilterSpec `json:"spec"`
Spec SriovResourcePolicySpec `json:"spec"`
}

// SriovResourceFilterSpec is the spec for a SriovResourceFilter
type SriovResourceFilterSpec struct {
// SriovResourcePolicySpec is the spec for a SriovResourcePolicy
type SriovResourcePolicySpec struct {
NodeSelector map[string]string `json:"nodeSelector,omitempty"`
Configs []Config `json:"configs,omitempty"`
}
Expand All @@ -60,9 +60,9 @@ type ResourceFilter struct {
// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// SriovResourceFilterList contains a list of SriovResourceFilter
type SriovResourceFilterList struct {
// SriovResourcePolicyList contains a list of SriovResourcePolicy
type SriovResourcePolicyList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []SriovResourceFilter `json:"items"`
Items []SriovResourcePolicy `json:"items"`
}
30 changes: 15 additions & 15 deletions pkg/api/sriovdra/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading