Implement RDMA subsystem mode change #666

e0ne · 2024-03-24T16:39:48Z

Now it's possible to configure RDMA subsystem mode using SR-IOV Network Operator in systemd mode.

We can't configure RDMA subsystem in a daemon mode because it should be done on host before any network namespace is created.

github-actions · 2024-03-24T16:39:58Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-03-24T16:51:12Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-03-24T17:30:04Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-03-24T17:45:50Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-03-24T18:09:50Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-03-24T20:26:39Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

coveralls · 2024-03-24T20:32:24Z

Pull Request Test Coverage Report for Build 11229466077

Details

84 of 189 (44.44%) changed or added relevant lines in 11 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.1%) to 45.063%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/host/internal/lib/netlink/netlink.go	0	3	0.0%
pkg/daemon/writer.go	0	6	0.0%
pkg/host/internal/network/network.go	19	25	76.0%
controllers/sriovnetworknodepolicy_controller.go	0	7	0.0%
pkg/daemon/daemon.go	1	8	12.5%
api/v1/zz_generated.deepcopy.go	2	12	16.67%
pkg/helper/mock/mock_helper.go	0	21	0.0%
pkg/host/mock/mock_host.go	0	21	0.0%
controllers/helper.go	50	74	67.57%

Files with Coverage Reduction	New Missed Lines	%
pkg/host/internal/network/network.go	2	57.56%

Totals
Change from base Build 11212485228:	-0.1%
Covered Lines:	6700
Relevant Lines:	14868

💛 - Coveralls

ykulazhenkov · 2024-03-25T07:58:58Z

pkg/systemd/systemd.go

 	newState.Spec.DpConfigVersion = ""

+	// shared mode is a default on OS
+	rdmaMode := consts.RdmaSubsystemModeShared


I think we should try to query/change mode only in case if rdmaMode parameter is explicitly set in the poolConfig, to provide a safer behavior for ENVs which doesn't use RDMA.

ykulazhenkov · 2024-03-25T08:00:19Z

cmd/sriov-network-config-daemon/service.go

 	hostHelpers.TryEnableTun()
 	hostHelpers.TryEnableVhostNet()

+	rdmaSubsystem, err := hostHelpers.GetRDMASubsystem()


I think we should execute this logic only if mode configuration is explicitly requested by a user.

ykulazhenkov · 2024-03-25T08:01:10Z

api/v1/sriovnetworkpoolconfig_types.go

+
+	// +kubebuilder:validation:Enum=shared;exclusive
+	// RDMA subsystem. Allowed value "shared", "exclusive".
+	RdmaMode string `json:"rdmaMode,omitempty"`


This option is only valid for systemd mode?
Do we want to document this somehow?

done as log message in a SriovNetworkPoolConfig controller

@e0ne cant we set this using module parameter ?

https://github.com/torvalds/linux/blob/55027e689933ba2e64f3d245fb1ff185b3e7fc81/drivers/infiniband/core/device.c#L128C1-L128C19

github-actions · 2024-03-25T20:11:11Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-03-25T20:39:21Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

ykulazhenkov

I added few additional comments

ykulazhenkov · 2024-03-26T08:12:34Z

cmd/sriov-network-config-daemon/service.go

+	if conf.RdmaMode != "" {
+		rdmaSubsystem, err := hostHelpers.GetRDMASubsystem()
+		if err != nil {
+			setupLog.Error(err, "failed to get RDMA subsystem mode")


If conf.RdmaMode is not empty string, then the user explicitly requested RDMA mode configuration. I think we can return error in this case. WDYT?

ykulazhenkov · 2024-03-26T08:13:11Z

cmd/sriov-network-config-daemon/service.go

+		if rdmaSubsystem != conf.RdmaMode {
+			err = hostHelpers.SetRDMASubsystem(conf.RdmaMode)
+			if err != nil {
+				setupLog.Error(err, "failed to set RDMA subsystem mode")


Do we want to return error here?

we want, thanks!

ykulazhenkov · 2024-03-26T08:14:48Z

pkg/host/internal/kernel/kernel.go


+func (k *kernel) GetRDMASubsystem() (string, error) {
+	log.Log.Info("GetRDMASubsystem(): retrieving RDMA subsystem mode")
+	chrootDefinition := utils.GetChrootExtension()


We have helper to enter chroot (part of utilsHelper). Do we want to use it here?

We'd got the same implementation in all `kernel' methods. Let's do it in a scope of a separate PR

github-actions · 2024-03-26T16:04:12Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-04-03T14:53:48Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-04-11T12:20:29Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-04-11T14:43:32Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

e0ne · 2024-04-15T07:02:42Z

@SchSeba could you please review this PR?

github-actions · 2024-06-25T09:33:49Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

d

adrianchiris · 2024-06-25T11:08:21Z

pkg/utils/cluster.go

 	return AnnotateObject(ctx, node, key, value, c)
 }
+
+func FindNodePoolConfig(ctx context.Context, node *corev1.Node, c client.Client) (*sriovnetworkv1.SriovNetworkPoolConfig, []corev1.Node, error) {


nit: add docstring

also im thinking we should have two functions:

find node pool for node

find nodes for node pool (with special handling for case where default node pool was provided)

WDYT ?

Also please add UT for whatever we end up with

adrianchiris · 2024-06-25T11:09:50Z

pkg/utils/cluster.go


+var (
+	oneNode     = intstr.FromInt32(1)
+	defaultNpcl = &sriovnetworkv1.SriovNetworkPoolConfig{Spec: sriovnetworkv1.SriovNetworkPoolConfigSpec{


nit: can we use full name here ? e.g defaultPoolConfig ?

also the 'l' at the end is not related

adrianchiris · 2024-06-25T11:24:19Z

pkg/utils/cluster.go

+			return nil, nil, err
+		}
+
+		// list all the nodes that are also part of this pool and return them


for those nodes why arent we validating they match exactly one ncp ? like in L223

adrianchiris · 2024-06-25T11:38:15Z

pkg/daemon/daemon.go

 	if vars.UsingSystemdMode {
 		log.Log.V(0).Info("nodeStateSyncHandler(): writing systemd config file to host")
+		// get node object
+		node := &corev1.Node{}


i dont see node is being used in this scope.

SchSeba · 2024-07-15T11:57:38Z

bindata/manifests/daemon/daemonset.yaml

              mountPath: /host/etc/os-release
              readOnly: true
        {{- end }}
+        {{- if .RDMACNIImage }}


Hi please rebase this PR now that we merged the rdma-cni deployment

SchSeba · 2024-07-15T11:58:54Z

cmd/sriov-network-config-daemon/service.go

 	hostHelpers.TryEnableTun()
 	hostHelpers.TryEnableVhostNet()

+	if conf.Spec.System.RdmaMode != "" {


I think we can remove this one as we do the configure via the modeprobe file

SchSeba · 2024-07-15T12:00:24Z

api/v1/sriovnetworknodestate_types.go

 }

+type System struct {
+	RdmaMode string `json:"rdmaMode,omitempty"`


I think you can add here also

// +kubebuilder:validation:Enum=shared;exclusive // RDMA subsystem. Allowed value "shared", "exclusive".

SchSeba · 2024-07-15T12:01:35Z

controllers/sriovnetworknodepolicy_controller.go

 		ns.Name = node.Name
 		ns.Namespace = vars.Namespace
 		j, _ := json.Marshal(ns)
+		netPoolConfig, _, err := utils.FindNodePoolConfig(context.Background(), &node, r.Client)


please use the context from the function don't create a new one

SchSeba · 2024-07-15T12:02:57Z

controllers/sriovnetworkpoolconfig_controller.go

 	}

+	// RdmaMode could be set in systemd mode only
+	if instance.Spec.RdmaMode != "" {


please remove this one as we support this on both modes

SchSeba · 2024-07-15T12:04:43Z

pkg/host/internal/kernel/kernel.go

 	return nil
 }

+func (k *kernel) DiscoverRDMASubsystem() (string, error) {


I think we can move this function to the network or sriov package

SchSeba · 2024-07-15T12:05:06Z

pkg/host/internal/kernel/kernel.go


+func (k *kernel) DiscoverRDMASubsystem() (string, error) {
+	log.Log.Info("DiscoverRDMASubsystem(): retrieving RDMA subsystem mode")
+	subsystem, err := netlink.RdmaSystemGetNetnsMode()


please use the netlink interface in the project so we can have a mock for it on unit tests

SchSeba · 2024-07-15T12:05:27Z

pkg/host/internal/kernel/kernel.go

+	return subsystem, nil
+}
+
+func (k *kernel) SetRDMASubsystem(mode string) error {


I think this function is no needed now that we use the modprobe file

SchSeba · 2024-07-15T12:06:13Z

pkg/utils/cluster.go

 	return AnnotateObject(ctx, node, key, value, c)
 }
+
+func FindNodePoolConfig(ctx context.Context, node *corev1.Node, c client.Client) (*sriovnetworkv1.SriovNetworkPoolConfig, []corev1.Node, error) {


SchSeba · 2024-08-27T19:48:00Z

Hi @e0ne can you please rebase the PR?

e0ne · 2024-08-27T19:51:10Z

Hi @e0ne can you please rebase the PR?

done

SchSeba

nice work!

I left some small comments

SchSeba · 2024-09-01T15:03:52Z

controllers/drain_controller.go

-		}
-		return defaultNpcl, defaultNodeLists, nil
-	}
+	return utils.FindNodePoolConfig(ctx, node, dr.Client)


should we put this in the helper of the controllers?
I don't want to utils to start growing again

it makes sense. done

SchSeba · 2024-09-01T16:22:14Z

controllers/sriovnetworknodepolicy_controller.go

 		ns.Name = node.Name
 		ns.Namespace = vars.Namespace
 		j, _ := json.Marshal(ns)
+		netPoolConfig, _, err := utils.FindNodePoolConfig(ctx, &node, r.Client)


just a general todo here we should have in memory map I think for this

could you please elaborate on this?

SchSeba · 2024-09-01T16:23:46Z

pkg/host/internal/network/network.go

 	"github.com/k8snetworkplumbingwg/sriov-network-operator/pkg/vars"
 )

+var ManifestsPath = "./bindata/manifests"


lets put this in consts

It's not needed anymore, so I deleted it

pkg/host/internal/network/network.go

SchSeba · 2024-09-01T16:25:48Z

pkg/host/internal/network/network.go

+		modeValue = 0
+	}
+	config := fmt.Sprintf("options ib_core netns_mode=%d\n", modeValue)
+	err := os.WriteFile("/etc/modprobe.d/ib_core.conf", []byte(config), 0644)


please use the getExtention here so we know if we are not inside a chroot

SchSeba · 2024-09-01T16:26:13Z

pkg/host/internal/network/network.go

+		return fmt.Errorf("failed to write ib_core config: %v", err)
+	}
+
+	err = os.WriteFile(path.Join(consts.Chroot, "/etc/modprobe.d/ib_core.conf"), []byte(config), 0644)


this looks like a duplicate?

rebase issue, it's deleted now

SchSeba · 2024-09-01T16:27:11Z

pkg/render/render.go

 	return out, nil
 }

+func RenderToString(path string, d *RenderData) (string, error) {


I don't think we need this function where we use it?

SchSeba · 2024-09-01T16:28:01Z

pkg/utils/cluster.go

 	return AnnotateObject(ctx, node, key, value, c)
 }
+
+func FindNodePoolConfig(ctx context.Context, node *corev1.Node, c client.Client) (*sriovnetworkv1.SriovNetworkPoolConfig, []corev1.Node, error) {


please move this function to the helpers in controllers better then adding more stuff to utils

adrianchiris · 2024-10-10T12:06:18Z

controllers/sriovnetworknodepolicy_controller.go

 		ns.Name = node.Name
 		ns.Namespace = vars.Namespace
 		j, _ := json.Marshal(ns)
+		netPoolConfig, _, err := findNodePoolConfig(ctx, &node, r.Client)


nit: move this b4 L274 so j contains rdmamode information ?

adrianchiris · 2024-10-10T12:06:44Z

controllers/sriovnetworknodepolicy_controller.go

 		j, _ := json.Marshal(ns)
+		netPoolConfig, _, err := findNodePoolConfig(ctx, &node, r.Client)
+		if err != nil {
+			log.Log.Error(err, "nodeStateSyncHandler(): failed to get SriovNetworkPoolConfig for the current node")


nit: err msg func name is wrong

adrianchiris · 2024-10-10T12:14:11Z

pkg/host/internal/lib/netlink/netlink.go

 	// IsLinkAdminStateUp checks if the admin state of a link is up
 	IsLinkAdminStateUp(link Link) bool
+	// DiscoverRDMASubsystem returns RDMA subsystem mode
+	DiscoverRDMASubsystem() (string, error)


nit: any chance to stick to the method name from netlink lib ?(RdmaSystemGetNetnsMode)

adrianchiris · 2024-10-10T12:20:44Z

pkg/host/internal/network/network.go

+	return subsystem, nil
+}
+
+func (n *network) SetRDMASubsystem(mode string) error {


should we make the distinction between: (?)

mode is "shared"

mode is "exclusive"

mode is unspecified (i.e "") which means system default

the latter would mean we need to delete the file.

changing the default value in kernel is a matter of one line change:
https://github.com/torvalds/linux/blob/d3d1556696c1a993eec54ac585fe5bf677e07474/drivers/infiniband/core/device.c#L127

adrianchiris · 2024-10-10T12:23:06Z

pkg/host/internal/network/network.go

+		modeValue = 0
+	}
+	config := fmt.Sprintf("options ib_core netns_mode=%d\n", modeValue)
+	path := filepath.Join(vars.FilesystemRoot, consts.Host, "etc", "modprobe.d", "ib_core.conf")


maybe use a more unique name ? e.g sriov_network_operator_modules_config.conf
ib_core.conf feels like a file that might exists with some values that we override when re-writing it.

also i wonder if we should search all conf files and see if the value is already there or we have a conflict and log it.

generally we dont expect this module parameter to be specified in the system.

adrianchiris · 2024-10-10T12:33:09Z

pkg/host/internal/network/network.go

+	if mode == "exclusive" {
+		modeValue = 0
+	}
+	config := fmt.Sprintf("options ib_core netns_mode=%d\n", modeValue)


perhaps add some comment to the beginning of the file like

# This file is managed by sriov-network-operator do not edit.

adrianchiris · 2024-10-10T12:44:27Z

pkg/daemon/daemon.go

 		reqReboot = reqReboot || r
 	}

+	if dn.currentNodeState.Status.System.RdmaMode != dn.desiredNodeState.Spec.System.RdmaMode {


we need to handle the case when dn.desiredNodeState.Spec.System.RdmaMode is empty (system default)

in this case need to delete the file if its present and decide if reboot is needed depending on the current kernel default.

root# modinfo ib_core filename: /lib/modules/5.15.0-121-generic/kernel/drivers/infiniband/core/ib_core.ko alias: rdma-netlink-subsys-4 license: Dual BSD/GPL description: core kernel InfiniBand API author: Roland Dreier alias: net-pf-16-proto-20 alias: rdma-netlink-subsys-5 srcversion: C45D89EC6DCCFE96001D79F depends: retpoline: Y intree: Y name: ib_core vermagic: 5.15.0-121-generic SMP mod_unload modversions sig_id: PKCS#7 signer: Build time autogenerated kernel key sig_key: 5E:7B:57:CA:17:D7:74:58:75:3F:84:AD:DE:07:46:5C:DC:AD:16:4E sig_hashalgo: sha512 signature: AE:90:AA:07:BB:6C:07:8C:AD:25:51:4B:1A:C6:FC:9F:D1:14:5B:B9: 90:F0:F5:84:E6:85:10:7E:AD:79:B5:04:5E:38:CF:5F:EC:6C:CD:BD: E5:BD:4D:4A:5D:7F:76:56:5E:DA:F0:C3:EA:63:98:0A:EE:B8:51:06: 42:8F:FD:08:51:28:DC:AD:4A:38:2E:A4:C4:7C:9E:42:4F:37:98:AD: 4D:8F:7F:5C:5C:41:93:27:62:C2:A1:D8:A0:5E:D5:15:25:5A:B9:C6: 8C:4D:17:CC:1F:A1:72:FE:18:5C:08:55:64:E6:A2:A7:2C:DD:57:1D: 03:A1:8C:12:17:76:61:72:E7:F9:A4:8F:F9:26:8F:36:02:8F:C6:56: 7B:A4:9E:6D:1D:ED:28:0E:7A:B5:81:F2:F0:FC:C4:05:0F:37:44:D3: C6:F4:00:B9:81:E2:32:EB:9B:1B:8E:EF:E5:CA:73:8F:4D:5E:11:80: 51:80:EB:AD:EC:97:2D:30:15:E9:8F:6B:9B:DB:40:5F:89:99:94:B1: 01:16:82:EF:22:01:5A:0F:14:F2:DE:64:68:76:3F:8B:26:F5:E9:97: E3:7F:DD:23:18:B2:A6:8F:8F:0F:A2:74:E1:B0:18:9F:E0:46:9F:7A: BE:89:9C:B7:C6:D4:47:64:70:E9:28:69:DC:A1:B0:F9:CB:A3:84:67: DF:68:A3:3D:E5:93:63:7D:91:A4:86:A9:CC:AA:DA:08:A8:64:97:D5: CC:BB:13:BB:28:17:87:1B:10:1B:2C:43:A6:0D:A0:05:6F:DB:45:03: 1C:0B:C5:67:37:94:CB:E3:CB:CF:03:6F:81:80:F2:77:E1:FD:09:2A: 8F:0F:FE:EA:C0:B8:CD:14:D2:69:55:0F:2F:82:3D:2D:30:0B:6E:72: 42:0C:F4:AB:6C:F8:D4:CA:45:AF:74:C9:A1:5D:EC:BE:C6:8C:81:4B: 2F:F4:46:EE:F6:28:83:11:B5:0D:EE:38:53:68:EF:1E:AC:AC:A9:B0: 91:C6:76:D4:46:2E:DA:CB:47:66:99:42:84:E2:31:99:35:C2:A5:4B: 04:F8:6A:34:E7:8A:AA:76:F3:83:DF:A8:82:E9:C8:14:05:51:90:F3: 18:31:3D:A7:40:F8:EE:32:B9:F7:C2:01:9F:71:2A:B1:8C:00:34:0F: F2:7C:DE:50:54:E3:CF:4B:EA:05:43:AF:E3:9D:A1:05:E6:A8:48:EE: 82:B7:6B:06:E3:C5:3D:AA:48:92:63:D8:7B:54:3E:F4:45:C7:5B:F6: 77:97:DD:32:93:ED:AC:DB:AD:EB:24:81:89:24:4F:25:A8:34:EA:63: A1:D4:FC:D8:B2:B2:41:61:C3:D3:E3:F5 parm: send_queue_size:Size of send queue in number of work requests (int) parm: recv_queue_size:Size of receive queue in number of work requests (int) parm: netns_mode:Share device among net namespaces; default=1 (shared) (bool) parm: force_mr:Force usage of MRs for RDMA READ/WRITE operations (bool)

maybe parse the cmd above for:
parm: netns_mode:Share device among net namespaces; default=1 (shared) (bool)

for now the default is shared, i dont know how likely its to change. maybe we can assume shared is the default.

SchSeba

in general another point we need to take care here is when we create/update a pool it will not do anything we need to wait for the nodePolicy controller to apply this.

we have two options here:

we pass a channel so the pool controller can trigger a policy one
we handle the system section in the pool controller directly

I must say I am not sure what is the best option @adrianchiris @zeeke WDYT?

SchSeba · 2024-10-10T15:02:38Z

/hold

this doesn't work on OCP it puts the node in boot loop because the mode didn't change.

adrianchiris · 2024-10-10T15:12:02Z

in general another point we need to take care here is when we create/update a pool it will not do anything we need to wait for the nodePolicy controller to apply this.

we have two options here:

we pass a channel so the pool controller can trigger a policy one

we handle the system section in the pool controller directly

I must say I am not sure what is the best option @adrianchiris @zeeke WDYT?

we can watch on pool obj as well and trigger reconcile event.

adrianchiris · 2024-10-10T15:13:29Z

this doesn't work on OCP it puts the node in boot loop because the mode didn't change.

writing files under /etc/modprobe.d is not persistent in coreOS ?

SchSeba · 2024-10-10T16:52:16Z

we can watch on pool obj as well and trigger reconcile event.

sure that also can work on we don't expect machine changes on it

SchSeba · 2024-10-10T18:15:06Z

writing files under /etc/modprobe.d is not persistent in coreOS ?

I checked with our kernel team on OCP platform.
the only way to do it for OCP will be to add it to the kernel args like we do with iommu :(
we can't use the etc/modprobe.d files

@e0ne let me know if you want me to work on this and push the changes for you

SchSeba · 2024-10-31T14:18:08Z

closing this one in favor or #799

e0ne force-pushed the rdma-subsytem-mode branch from c986251 to d3641ab Compare March 24, 2024 16:51

e0ne force-pushed the rdma-subsytem-mode branch from d3641ab to a0af988 Compare March 24, 2024 17:29

e0ne force-pushed the rdma-subsytem-mode branch from a0af988 to a3ce3a3 Compare March 24, 2024 17:45

e0ne force-pushed the rdma-subsytem-mode branch from a3ce3a3 to 3952e04 Compare March 24, 2024 18:09

e0ne force-pushed the rdma-subsytem-mode branch from 3952e04 to bcb0804 Compare March 24, 2024 20:26

ykulazhenkov reviewed Mar 25, 2024

View reviewed changes

e0ne force-pushed the rdma-subsytem-mode branch from bcb0804 to 79013d7 Compare March 25, 2024 20:11

e0ne force-pushed the rdma-subsytem-mode branch from 79013d7 to f60fdf7 Compare March 25, 2024 20:39

ykulazhenkov reviewed Mar 26, 2024

View reviewed changes

e0ne force-pushed the rdma-subsytem-mode branch from f60fdf7 to a5f0d3b Compare March 26, 2024 16:04

e0ne force-pushed the rdma-subsytem-mode branch from a5f0d3b to feb4bd0 Compare April 3, 2024 14:53

e0ne force-pushed the rdma-subsytem-mode branch from feb4bd0 to ab92392 Compare April 11, 2024 12:20

e0ne force-pushed the rdma-subsytem-mode branch from ab92392 to f37c6d1 Compare April 11, 2024 14:43

e0ne force-pushed the rdma-subsytem-mode branch from f37c6d1 to 3d19033 Compare April 16, 2024 15:05

e0ne force-pushed the rdma-subsytem-mode branch from b032b21 to e0ae318 Compare June 25, 2024 09:33

adrianchiris previously approved these changes Jun 25, 2024

View reviewed changes

adrianchiris reviewed Jun 25, 2024

View reviewed changes

SchSeba reviewed Jul 15, 2024

View reviewed changes

e0ne force-pushed the rdma-subsytem-mode branch from e0ae318 to 8ce9b72 Compare August 27, 2024 18:47

e0ne force-pushed the rdma-subsytem-mode branch 2 times, most recently from e294415 to 288e028 Compare August 28, 2024 08:34

SchSeba reviewed Sep 1, 2024

View reviewed changes

e0ne force-pushed the rdma-subsytem-mode branch from 288e028 to 596e73d Compare September 9, 2024 11:33

e0ne requested a review from SchSeba September 9, 2024 11:34

e0ne force-pushed the rdma-subsytem-mode branch from 596e73d to 4e5c92b Compare October 7, 2024 20:07

RDMA subsystem is implemented via ib_core module config.

60432e0

e0ne force-pushed the rdma-subsytem-mode branch from 4e5c92b to 60432e0 Compare October 8, 2024 05:59

e0ne requested a review from adrianchiris October 8, 2024 06:00

adrianchiris reviewed Oct 10, 2024

View reviewed changes

SchSeba reviewed Oct 10, 2024

View reviewed changes

github-actions bot added the hold label Oct 10, 2024

rollandf mentioned this pull request Oct 22, 2024

bug: rdma exlusive handling k8snetworkplumbingwg/sriov-network-device-plugin#603

Merged

SchSeba closed this Oct 31, 2024

e-minguez mentioned this pull request Aug 7, 2025

RDMA cannot be disabled #919

Open

Implement RDMA subsystem mode change #666

Implement RDMA subsystem mode change #666

Uh oh!

Conversation

e0ne commented Mar 24, 2024

Uh oh!

github-actions bot commented Mar 24, 2024

Uh oh!

github-actions bot commented Mar 24, 2024

Uh oh!

github-actions bot commented Mar 24, 2024

Uh oh!

github-actions bot commented Mar 24, 2024

Uh oh!

github-actions bot commented Mar 24, 2024

Uh oh!

github-actions bot commented Mar 24, 2024

Uh oh!

coveralls commented Mar 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 11229466077

Details

💛 - Coveralls

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 25, 2024

Uh oh!

github-actions bot commented Mar 25, 2024

Uh oh!

ykulazhenkov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 26, 2024

Uh oh!

github-actions bot commented Apr 3, 2024

Uh oh!

github-actions bot commented Apr 11, 2024

Uh oh!

github-actions bot commented Apr 11, 2024

Uh oh!

e0ne commented Apr 15, 2024

Uh oh!

github-actions bot commented Jun 25, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

coveralls commented Mar 24, 2024 •

edited

Loading