Unclear reason of validating GPU count == number of NVLink partitions for instance

[prazumovsky]

Regarding

bare-metal-manager-rest/api/pkg/api/model/instance.go

Line 328 in e65b165

"nvLinkInterfaces": errors.New("number of NVLink Interfaces must match the number of GPU indexes"),

- why the validation function requests for the number of nvlink partitions == the number of GPUs presented in machine caps?

What if I want to add only 2 out of 4 GPUs to partition for an instance? For example, I'm creating an instance and it becomes ready. Then I create nvlink logical partition via API and update the instance with assigning 2 GPUs to this partition. Then API will return this exact error because the rest 2 GPUs are not presented in instance's nvlink interface. As far as I know, nvlink partitioning supports creating partitions only for 1 GPU so it should be the same for 2 GPUs in partition.

One more small question:
I see that nvlink interface supports gpuGuid:

bare-metal-manager-rest/api/pkg/api/model/nvlinkinterface.go

Line 73 in e65b165

GpuGUID *string `json:"gpuGuid"`

but in API docs I see only deviceInstance to use in nvlink interface definition for update instance: https://nvidia.github.io/bare-metal-manager-rest/#tag/Instance/operation/update-instance. May we use gpuGuid instead of deviceInstance? If not then how we can get deviceInstance value for GPUs in bare-metal manager API?

[thossain-nv]

Hi Peter, the restriction was put in place with the assumption was the user would want all their GPUs connected. However it seems that in your use case you only want 2 of the GPUs connected and connecting the other 2 is not necessary.

If the user doesn't want all their GPUs connected to Partitions, I don't see why we need to force them to do so. We can remove this restriction. Please note that the NVLink Interfaces data sent when updating Instances replaces existing NVLink Interfaces.

Regarding specifying GPU GUID - when Instances are created using Instance Types, Machines are chosen at random from the Instance Type pool, so we cannot know in advance which exact Machine will be chosen and what GUIDs it would have. When creating Instances by specifying Machine ID, it may be possible to accept GUID but we'd want to keep the interface consistent.

If you are creating Instances by specifying Machine ID, then you can fetch Machine detail including metadata and view the GPU information. deviceInstance is the index we get when GPUs are sorted by their slot number so GPU GUID can be correlated to device instances.

[prazumovsky]

If the user doesn't want all their GPUs connected to Partitions, I don't see why we need to force them to do so. We can remove this restriction. Please note that the NVLink Interfaces data sent when updating Instances replaces existing NVLink Interfaces.

That would be great!

I see why this validation could be possibly added. Here is a description that if no user partition is created for instance, BMM creates default one and adds all GPUs to that partition: https://github.com/NVIDIA/bare-metal-manager-core/blob/d2ee2785cc207b3e125171d1cd6ba1411ed4d658/docs/nvlink.md?plain=1#L29. Thus instance should have partition with all GPUs by default and then if we want to push GPUs to another partition - we would still have all gpus attached to nvlink interfaces: N in user partition and M in default where N+M=GPUs total. So the state meets this requirement.

(However, for some reason it is not true for my environment, here is a bug about it: NVIDIA/bare-metal-manager-core#303)

Overall, I don't see a reason to limit user of splitting GPUs to multiple partitions per compute tray. For example, I have a kubernetes cluster and pushing GPUs inside kube workloads in host passthrough mode. I want splitting 2 GPUs per workload and isolate them from each other. These workloads are not creating at one time - thus I need only 2 GPUs to define in partition at the time. So I'd be appreciate to drop hard requirement of always using all GPUs in nvlink interfaces per instance.

[thossain-nv]

@prazumovsky We will work on fixing the doc. Here is the actual logic:

• User can set a default NVLink Logical Partition for a VPC. This is not done automatically - user can set the value when creating or updating the VPC. The default cannot be set by updating the VPC if it already contains Instances. When user creates an Instance under a VPC with default NVLink Logical Partition set, all GPUs for the Instance will be automatically connected to the default Partition, no need to specify NVLink Interfaces. If NVLink Interfaces are specified, the must all specify the default Partition.
• If user wants fine grained NVLink Logical Partitioning for their Instance GPUs, they should not set a default NVLink Logical Partition for the VPC. Instead they should explicitly specify NVLink Interfaces when creating/updating Instance and each of the Interfaces can connect to a different NVLink Logical Partition if desired. The restriction I described above applies to this case. User can specify different Partitions for the GPUs of the same Instance, but they have to specify them for all GPUs, there is no default here. If we lift the restriction, and user specifies 2 Interfaces out of 4, 2 of them won't be connected to any Partition.

Sorry if this seems complicated, basically we have "set VPC default for ease" or "advanced configuration".

[nikitamarchenko]

Hi, how we can setup nvlink interfaces on instance creation? We don't know how many gpus on machine before allocation.
Is valid steps for now is >

1. create instance
2. get nvlink interfaces from machine endpoint.
3. patch instance

Thank you.

[thossain-nv]

Hi @nikitamarchenko, we recommend creating Instance Types with the appropriate capabilities. Here is an example of an Instance Type that has InfiniBand and NVLink capable GPUs. When created, only Machines that match these capabilities can be assigned to the Instance Type. When Tenant is creating Instances with this Instance Type, they can be aware of the deviceInstance values they can use by inspecting the Instance Type capabilities.

curl -X POST https://<domain>/v2/org/$ORG_NAME/carbide/instance/type \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -d @- << EOF
{
  "name": "gpu.gb200.ib",
  "siteId": "$SITE_ID",
  "machineCapabilities": [
    {
      "type": "InfiniBand",
      "name": "MT2910 Family [ConnectX-7]",
      "count": 4
    },
    {
      "type": "GPU",
      "name": "NVIDIA GB200",
      "count": 4,
      "deviceType": "NVLink"  
    }   
  ]
}
EOF

[prazumovsky]

@thossain-nv I continued to research the case I need and found that the API has another validation which is blocking my case of allocating only 2 GPUs of an instance into the partition:

400 Bad Request: {"source":"forge","message":"Failed to validate Instance update request data","data":{"nvLinkInterfaces":"all the NVLink Interfaces must have same NVLink Logical Partition"}

bare-metal-manager-rest/api/pkg/api/model/instance.go

Lines 509 to 511 in 4d39f29

	return validation.Errors{
	"nvLinkInterfaces": errors.New("all the NVLink Interfaces must have same NVLink Logical Partition"),
	}

IMO I don't see any valuable reason to forbid splitting instance's GPUs on, for example, 2 separate partitions to run 2 separate workloads which requires only 2 GPUs to be completely done. Again, NVSwitch user partitions as it said in documentation allows to create up to 72 partitions with 1 partition per GPU.

[thossain-nv]

Hi @prazumovsky wanted to provide an update that we're working on verifying that the following cases would execute e2e without any issues:

• Specifying NVLink Interfaces for a subset of GPUs of an Instance
• Specifying different NVLink Partition for different NVLink Interfaces of an Instance
  I'll be adding a PR in the next day or two to remove these restrictions from the API endpoint. We'll merge once we have confirmed the e2e tests.

[rtamma-nv]

NVLink partition monitor in carbide should be able to handle the above two cases, but we have not tested these scenarios on real HW. Will will run some tests to verify that it is functioning properly and confirm.

[prazumovsky]

@thossain-nv @rtamma-nv thank you!