Cleanup Multi Node API to avoid redundant GPU input by visheshtanksale · Pull Request #631 · NVIDIA/k8s-nim-operator

visheshtanksale · 2025-08-22T22:05:37Z

Removed field spec.multiNode.GPUSPerPod
The GPUs per node will be determined by the gpu specified in the spec.Resources
Added webhook validation to check that GPU values is set on spec.Resources
TODO: if GPU is not specified on spec.Resources then need to auto determine base on tp*pp/(.spec.multiNode.size)

copy-pr-bot · 2025-08-22T22:05:40Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

varunrsekar · 2025-08-22T23:44:05Z

api/apps/v1alpha1/nimservice_types.go


+// GetMultiNodeGPUsPerPod returns the number of GPUs per pod for the multi-node NIMService.
+func (n *NIMService) GetMultiNodeGPUsPerPod() int {
+	gpuQuantity, ok := n.Spec.Resources.Requests["nvidia.com/gpu"]


do we need to support DRA here?

Added DRA support. I am assuming for ResourceClaim and ResourceClaimTemplate that there might be only one request for GPUs under Spec.Devices.Requests. Is this correct assumption?

internal/shared/multinode.go

shivamerla · 2025-08-25T15:59:29Z

internal/shared/resourceclaims.go

+// Constants for GPU device detection.
+const (
+	// NVIDIA identifiers used to detect GPU devices.
+	NVIDIAIdentifier    = "nvidia"


These might be used for non-gpu devices too for e.g. rdma.nvidia.com. We should just check for a well known device-class gpu.nvidia.com or if custom ones are set, then make sure the Cel expression exist in those classes

expression: "device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].type == 'gpu'"

Fixed this to look for gpu.nvidia.com

internal/controller/platform/kserve/nimservice.go

api/apps/v1alpha1/nimservice_types.go

internal/controller/platform/standalone/nimservice.go

api/apps/v1alpha1/nimservice_types.go

shengnuo · 2025-08-25T18:25:32Z

internal/shared/resourceclaims.go

+		switch {
+		case resource.ResourceClaimName != nil:
+			gpuCount, err = GetGPUDeviceCountForClaim(ctx, client, *resource.ResourceClaimName, namespace)
+		case resource.ResourceClaimTemplateName != nil:
+			gpuCount, err = GetGPUDeviceCountForClaimTemplate(ctx, client, *resource.ResourceClaimTemplateName, namespace)
+		case resource.ClaimCreationSpec != nil:
+			gpuCount, err = GetGPUDeviceCountForClaimCreationSpec(ctx, client, *resource.ClaimCreationSpec, namespace)
+		}


I am wondering do we have a use case for users to specify two of (resourceClaimName, resoruceClaimTemplateName, claimCreationSpec)...?

I thought users would only specify one of three?
If that's the case, we should consider adding to the validation webhook

@shengnuo this is possible across the spec.draResources[] array. 1 entry may have a resourceClaimName while another one may have a claimCreationSpec. We would need to aggregate them

Its possible for users to specify all three resource specification types

varunrsekar · 2025-08-25T18:56:24Z

internal/controller/platform/standalone/nimservice.go

 	// TODO auto determine base on tp*pp/(.spec.multiNode.size)
 	if nimService.Spec.MultiNode != nil {
-		gpuQuantity, err = apiResource.ParseQuantity(fmt.Sprintf("%d", nimService.Spec.MultiNode.GPUSPerPod))
+		gpuQuantity, err = apiResource.ParseQuantity(fmt.Sprintf("%d", multiNodeGPUsPerPod))


I'm confused why this change is needed. multiNodeGPUsPerPod is computed based on the count from nvidia.com/gpu resource requests/limits. It doesnt make sense to do it here which only executes when resource requests/limits is missing.

Removed this

varunrsekar · 2025-08-25T18:57:24Z

internal/shared/multinode.go

+// GetGPUCountPerPod returns the number of GPUs per pod for the NIMService.
+func GetGPUCountPerPod(ctx context.Context, client client.Client, nimService *appsv1alpha1.NIMService) (int, error) {
+
+	if nimService.Spec.DRAResources == nil {


nit

Suggested change

if nimService.Spec.DRAResources == nil {

if len(nimService.Spec.DRAResources) == 0 {

varunrsekar · 2025-08-25T19:00:09Z

internal/shared/multinode.go

+
+	if nimService.Spec.DRAResources == nil {
+		if nimService.Spec.Resources == nil {
+			return 0, nil


we should return an error when no GPUs are specified and fail the NIMService for multiNode.

varunrsekar · 2025-08-25T19:00:49Z

internal/shared/multinode.go

+		gpuQuantity, ok := nimService.Spec.Resources.Requests["nvidia.com/gpu"]
+		if !ok {
+			// return 0 if no GPU limit is specified because auto determine base on tp*pp/(.spec.multiNode.size) is a TODO
+			return 0, nil
+		}


can you also check for resource limits here?

internal/shared/resourceclaims.go

varunrsekar · 2025-08-25T19:04:12Z

internal/shared/resourceclaims.go

+	}
+
+	hint := strings.ToLower(fmt.Sprint(dc.Spec))
+	return strings.Contains(hint, NVIDIAIdentifier) || strings.Contains(hint, NVIDIAComIdentifier), nil


nit: we should restrict to gpu.nvidia.com device class.

varunrsekar · 2025-08-25T19:05:59Z

internal/shared/resourceclaims.go

+		switch {
+		case resource.ResourceClaimName != nil:
+			gpuCount, err = GetGPUDeviceCountForClaim(ctx, client, *resource.ResourceClaimName, namespace)
+		case resource.ResourceClaimTemplateName != nil:
+			gpuCount, err = GetGPUDeviceCountForClaimTemplate(ctx, client, *resource.ResourceClaimTemplateName, namespace)
+		case resource.ClaimCreationSpec != nil:
+			gpuCount, err = GetGPUDeviceCountForClaimCreationSpec(ctx, client, *resource.ClaimCreationSpec, namespace)
+		}


@shengnuo this is possible across the spec.draResources[] array. 1 entry may have a resourceClaimName while another one may have a claimCreationSpec. We would need to aggregate them

varunrsekar · 2025-08-25T19:08:18Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

-		}
+
+	// Validate that Claims must be empty
+	if spec.Resources != nil && spec.Resources.Claims != nil && len(spec.Resources.Claims) > 0 {


nit: if its nil, then len(...) will return 0

Suggested change

if spec.Resources != nil && spec.Resources.Claims != nil && len(spec.Resources.Claims) > 0 {

if spec.Resources != nil && len(spec.Resources.Claims) > 0 {

varunrsekar · 2025-08-25T19:08:43Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+	gpuResourceName := corev1.ResourceName("nvidia.com/gpu")
+
+	// Check if GPU requests are specified
+	if spec.Resources == nil || spec.Resources.Requests == nil {


nit: also check for resource limits

varunrsekar · 2025-08-25T19:09:56Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

 	errList = append(errList, validateMetricsConfiguration(&spec.Metrics, fldPath.Child("metrics"))...)
 	errList = append(errList, validateScaleConfiguration(&spec.Scale, fldPath.Child("scale"))...)
-	errList = append(errList, validateResourcesConfiguration(spec.Resources, fldPath.Child("resources"))...)
+	errList = append(errList, validateResourcesConfiguration(spec, fldPath.Child("resources"))...)


Suggested change

errList = append(errList, validateResourcesConfiguration(spec, fldPath.Child("resources"))...)

errList = append(errList, validateResourcesConfiguration(spec, fldPath)...)

nit: its better to match the fieldPath with the input in the caller.

shivamerla · 2025-08-26T15:34:45Z

internal/controller/nimservice_controller.go


 	logger.Info("Reconciling", "NIMService", nimService.Name)

+	if nimService.Spec.MultiNode != nil && nimService.Annotations != nil {


annotations could be nil initially, we would need to initialize the map for the first time.

also, this should be done in the platform specific reconciler where we render the LWS resource. That way we can consider the case of auto GPU assignment too based on the tp size in the optimized profile, otherwise there would be mismatch and we end up setting this annotation to 0.

Updated the annotation nil check.
Auto assign of GPU is currently not supported for Multi Node. We need to discuss this scenario.

shivamerla · 2025-08-26T15:41:04Z

internal/controller/platform/kserve/nimservice.go

+	// Get tensorParallelism from the profile
+	tensorParallelism, err := utils.GetTensorParallelismByProfileTags(profile.Config)
+	if err != nil {
+		logger.Error(err, "Failed to retrieve tensorParallelism")


Suggested change

logger.Error(err, "Failed to retrieve tensorParallelism")

logger.Error(err, "Missing nvidia.com/gpu resource request and unable to retrieve tensorParallelism for NIM profile")

shivamerla · 2025-08-26T15:51:18Z

internal/shared/resourceclaims.go

+
+		isGPU, err := isNVIDIAGPU(ctx, client, req.Exactly.DeviceClassName)
+		if err != nil {
+			// This allows partial success scenarios


we would need to fail in all cases here as GPU count might be incorrect, if there is an error getting the device-class for e.g.

shivamerla · 2025-08-26T15:59:29Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+	_, hasRequests := spec.Resources.Requests[gpuResourceName]
+	_, hasLimits := spec.Resources.Limits[gpuResourceName]
+
+	if !hasRequests && !hasLimits {


this will fail when an optimized profile is selected and we auto-retrieve GPUs based on the tp size.

This check is only for multi node NIM Service. Currently, we dont have support for multi node NIMService to assign GPU Resources automatically here. Since I have removed the .spec.multiNode.gpuPerWorker, user has to provide GPU resource limits for multi node

but we auto assign based on tp size for optimized profile as well with changes in the assignGPUResources call.

shengnuo · 2025-08-27T21:55:36Z

internal/controller/platform/kserve/nimservice.go

+	// If no user-provided GPU resource is found, proceed with auto-assignment
+	// Get tensorParallelism from the profile
+	tensorParallelism, err := utils.GetTensorParallelismByProfileTags(profile.Config)
+	if err != nil {
+		logger.Error(err, "Missing nvidia.com/gpu resource request/limit and unable to retrieve tensorParallelism for NIM profile")
+		return nil, err
+	}
+	if tensorParallelism != "" {
+		gpuQuantity, err = apiResource.ParseQuantity(tensorParallelism)


the addGPUResources function look exactly the same between standadlone and kserve controllers...

Consider adding it to internal/controller/shared

this can be done as a follow up as this applies to many functions between those two packages.

shivamerla · 2025-08-29T19:02:52Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+	gpuResourceName := corev1.ResourceName("nvidia.com/gpu")
+
+	// Check if GPU requests or limits are specified
+	if spec.Resources == nil {


This should also consider spec.DRAResources

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

shivamerla · 2025-09-02T15:43:17Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+		return errList
+	}
+
+	if spec.Resources != nil {


@visheshtanksale This could just include CPU/Memory requests, while spec.draResources can include GPU requests. First we need to check if draResources are specified then rely on spec.resources.

This check is primary on spec.resouces and we dont have any checks under spec.draResources because for that we need to make an API request. Since this includes only checks under spec.resources I feel it better to have the implementation here

shivamerla · 2025-09-02T15:49:04Z

api/apps/v1alpha1/nimservice_types.go


 	// +kubebuilder:default:=1
-	// Size specifies the number of pods to create for the multi-node NIMService.
+	// PipelineParallelism specifies the number of pods to create for the multi-node NIMService.


i am inclined to add a parallelism type for this if we need to add additional methods in the future.

parallelism: pp: 1 (default/minimum)

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

varunrsekar · 2025-08-27T15:08:58Z

internal/controller/nimservice_controller.go

+		if nimService.Annotations == nil {
+			nimService.Annotations = map[string]string{}
+		}
+		if _, ok := nimService.Annotations[utils.GPUCountPerPodAnnotationKey]; !ok {


nit: can you use GetGPUCountPerPod here? its already reading that annotation value

varunrsekar · 2025-09-02T22:59:27Z

api/apps/v1alpha1/nimservice_types.go

-	// GPUSPerPod specifies the number of GPUs for each instance. In most cases, this should match `resources.limits.nvidia.com/gpu`.
-	GPUSPerPod int `json:"gpusPerPod,omitempty"`
+	// Parallelism specifies the parallelism strategy for the multi-node NIMService.
+	Parallelism Parallelism `json:"parallelism,omitempty"`


Suggested change

Parallelism Parallelism `json:"parallelism,omitempty"`

Parallelism *Parallelism `json:"parallelism"`

I'd expect this to be a required field

varunrsekar · 2025-09-02T23:02:00Z

api/apps/v1alpha1/nimservice_types.go

+	// +kubebuilder:default:=1
+	// PP specifies the number of pods to create for the multi-node NIMService.
+	// +kubebuilder:validation:Minimum=1
+	PP int `json:"pp,omitempty"`


Suggested change

PP int `json:"pp,omitempty"`

Pipeline *uint32 `json:"pp,omitempty"`

can you call this Pipeline? PP doesnt sound good

varunrsekar · 2025-09-02T23:02:18Z

api/apps/v1alpha1/nimservice_types.go

 }

+type Parallelism struct {
+	// +kubebuilder:default:=1


we shouldnt have defaults here. This wouldnt work for multi-node anyway

varunrsekar · 2025-09-02T23:06:36Z

internal/shared/nimservice.go

+			}
+		}
+		return int(gpuQuantity.Value()), nil
+	} else {


style nit: you dont need an else block here.

varunrsekar · 2025-09-02T23:22:16Z

internal/shared/resourceclaims.go

+	hint := strings.ToLower(fmt.Sprint(dc.Spec))
+	return strings.Contains(hint, NVIDIAGPUComIdentifier), nil


this is kinda flaky because it may just contain the term anywhere in the spec. Its better to be safe and iterating over all the selectors and looking only at the expressions.

As a follow-up we can also be careful in only checking if it contains device.driver == "gpu.nvidia.com" OR has device.attributes["gpu.nvidia.com"] OR has device.capacity["gpu.nvidia.com"]

varunrsekar · 2025-09-02T23:24:53Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+
+	// Validate that Claims must be empty
+	if spec.Resources != nil && len(spec.Resources.Claims) > 0 {
+		errList = append(errList, field.Forbidden(fldPath.Child("claims"), "must be empty"))


Suggested change

errList = append(errList, field.Forbidden(fldPath.Child("claims"), "must be empty"))

errList = append(errList, field.Forbidden(fldPath.Child("resources").Child("claims"), "must be empty"))

varunrsekar · 2025-09-02T23:25:38Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+}
+
+// validateGPURequirements ensures that GPU resources are properly configured for MultiNode deployments.
+func validateGPURequirements(spec *appsv1alpha1.NIMServiceSpec, fldPath *field.Path) field.ErrorList {


nit: call this validateMultiNodeGPURequirements?

varunrsekar · 2025-09-02T23:28:51Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+		if hasRequests {
+			gpuRequests := spec.Resources.Requests[gpuResourceName]
+			if gpuRequests.IsZero() || gpuRequests.Value() <= 0 {
+				errList = append(errList, field.Invalid(fldPath.Child("requests").Child("nvidia.com/gpu"), gpuRequests.String(), "must be greater than 0"))


Suggested change

errList = append(errList, field.Invalid(fldPath.Child("requests").Child("nvidia.com/gpu"), gpuRequests.String(), "must be greater than 0"))

errList = append(errList, field.Invalid(fldPath.Child("requests").Key("nvidia.com/gpu"), gpuRequests.String(), "must be greater than 0"))

varunrsekar · 2025-09-02T23:29:30Z

internal/webhook/apps/v1alpha1/nimservice_webhook_validation_helper.go

+		if hasLimits {
+			gpuLimits := spec.Resources.Limits[gpuResourceName]
+			if gpuLimits.IsZero() || gpuLimits.Value() <= 0 {
+				errList = append(errList, field.Invalid(fldPath.Child("limits").Child("nvidia.com/gpu"), gpuLimits.String(), "must be greater than 0"))


Suggested change

errList = append(errList, field.Invalid(fldPath.Child("limits").Child("nvidia.com/gpu"), gpuLimits.String(), "must be greater than 0"))

errList = append(errList, field.Invalid(fldPath.Child("limits").Key("nvidia.com/gpu"), gpuLimits.String(), "must be greater than 0"))

shivamerla · 2025-09-05T20:45:25Z

can we close this as the API change was merged. We need a follow up on adding validations in the webhook.

visheshtanksale · 2025-09-05T22:34:33Z

Closing this. Follow with webhook validations for next release

visheshtanksale requested review from ArangoGutierrez, shengnuo, shivamerla, slu2011 and varunrsekar as code owners August 22, 2025 22:05

visheshtanksale force-pushed the cleanup-gpu-per-node branch from 71c6b62 to 72a326d Compare August 22, 2025 22:10

varunrsekar reviewed Aug 22, 2025

View reviewed changes

shivamerla reviewed Aug 25, 2025

View reviewed changes

internal/shared/multinode.go Outdated Show resolved Hide resolved

shivamerla reviewed Aug 25, 2025

View reviewed changes

internal/controller/platform/kserve/nimservice.go Outdated Show resolved Hide resolved

shivamerla reviewed Aug 25, 2025

View reviewed changes

api/apps/v1alpha1/nimservice_types.go Outdated Show resolved Hide resolved

shengnuo reviewed Aug 25, 2025

View reviewed changes

varunrsekar reviewed Aug 25, 2025

View reviewed changes

shivamerla reviewed Aug 26, 2025

View reviewed changes

shengnuo reviewed Aug 27, 2025

View reviewed changes

shivamerla reviewed Aug 29, 2025

View reviewed changes

visheshtanksale enabled auto-merge (squash) August 29, 2025 22:07

visheshtanksale added 9 commits August 29, 2025 15:08

Cleanup Multi Node API to avoid redundant GPU input

49f9acc

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Adding support for DRA

78f9a6f

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Updating incorrect check for DRA

200f3f2

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Fixing Unit test

6f41833

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Addressing review comments

15a8768

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Addressing review comments

98de7e5

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Addressing review comments

b7ada4f

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Remove logic to select GPUs for multinode

15ba40a

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Adding logic to check for first available device from Device Request

3d27f2c

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

visheshtanksale added 5 commits August 29, 2025 15:08

Addressing review comments

ae880b7

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Addressing review comments

7b24969

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Addressing review comments

9147b61

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Updating size to be pipelineparallelism

d38622e

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Updating size to be pipelineparallelism

1f53369

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

visheshtanksale force-pushed the cleanup-gpu-per-node branch from 155695b to 1f53369 Compare August 29, 2025 22:08

visheshtanksale added 2 commits August 30, 2025 05:47

Adding RBAC for deviceclasses

a8ce37e

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Merge branch 'main' into cleanup-gpu-per-node

aa56a04

shivamerla reviewed Sep 2, 2025

View reviewed changes

visheshtanksale added 3 commits September 2, 2025 20:44

Updating pipelineparallelism field

1188da9

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Updating pipelineparallelism field

1271327

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

Updating pipelineparallelism field

678a649

Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>

varunrsekar reviewed Sep 2, 2025

View reviewed changes

visheshtanksale closed this Sep 5, 2025

auto-merge was automatically disabled September 5, 2025 22:34
Pull request was closed

	if nimService.Spec.DRAResources == nil {
	if len(nimService.Spec.DRAResources) == 0 {

	if spec.Resources != nil && spec.Resources.Claims != nil && len(spec.Resources.Claims) > 0 {
	if spec.Resources != nil && len(spec.Resources.Claims) > 0 {

	errList = append(errList, validateResourcesConfiguration(spec, fldPath.Child("resources"))...)
	errList = append(errList, validateResourcesConfiguration(spec, fldPath)...)


		logger.Info("Reconciling", "NIMService", nimService.Name)

		if nimService.Spec.MultiNode != nil && nimService.Annotations != nil {

	logger.Error(err, "Failed to retrieve tensorParallelism")
	logger.Error(err, "Missing nvidia.com/gpu resource request and unable to retrieve tensorParallelism for NIM profile")

	Parallelism Parallelism `json:"parallelism,omitempty"`
	Parallelism *Parallelism `json:"parallelism"`

	PP int `json:"pp,omitempty"`
	Pipeline *uint32 `json:"pp,omitempty"`

		hint := strings.ToLower(fmt.Sprint(dc.Spec))
		return strings.Contains(hint, NVIDIAGPUComIdentifier), nil

	errList = append(errList, field.Forbidden(fldPath.Child("claims"), "must be empty"))
	errList = append(errList, field.Forbidden(fldPath.Child("resources").Child("claims"), "must be empty"))

	errList = append(errList, field.Invalid(fldPath.Child("requests").Child("nvidia.com/gpu"), gpuRequests.String(), "must be greater than 0"))
	errList = append(errList, field.Invalid(fldPath.Child("requests").Key("nvidia.com/gpu"), gpuRequests.String(), "must be greater than 0"))

Conversation

visheshtanksale commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Aug 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

visheshtanksale commented Aug 22, 2025 •

edited

Loading

visheshtanksale Sep 2, 2025 •

edited

Loading