Updating the replicas for kserve #738
Conversation
visheshtanksale
commented
Jan 22, 2026
visheshtanksale
commented
- Fixing Add support for scaling down NIMServices to 0 replicas #710
- Not adding replicas on InferenceService for Knative/serverless deployment mode
Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>
visheshtanksale
requested review from
ArangoGutierrez,
shengnuo,
shivamerla and
varunrsekar
as code owners
|
/ok to test a64260a |
| SchedulerName string `json:"schedulerName,omitempty"` | ||
| Metrics Metrics `json:"metrics,omitempty"` | ||
| // +kubebuilder:validation:Minimum=1 | ||
| // +kubebuilder:validation:Minimum=0 |
There was a problem hiding this comment.
For non-kserve NIMs, is it expected to be able to deploy it with 0 replicas?
There was a problem hiding this comment.
Based on the issue linked looks like the user want to spin it down to 0 to optimize GPU usage as we don't have scale-to-zero support yet. @visheshtanksale if we want to support this then we need to avoid trying to set model status as well and set the status as NotReady or introduce a new state.
There was a problem hiding this comment.
Both for standalone and kserve inferencePlatform, when the NIMService is scaled down to zero the status is set to NotReady. I think this state is good enough.
There was a problem hiding this comment.
Okay I remember now. We had already added support to avoid querying the model details if the deployment is scaled down. This change makes sense with that context.
| } | ||
| } | ||
|
|
||
| _ = deploymentMode // Use the variable to avoid unused warning |
There was a problem hiding this comment.
this is not used anyway, we can remove this.
| params.ScaleTarget = ptr.To(target) | ||
| } | ||
| } else { | ||
| params.MinReplicas = ptr.To[int32](0) |
There was a problem hiding this comment.
For serverless/knative mode, if auto-scaling is enabled, shouldn't we be setting predictor.minReplicas to either scale.minReplicas or the annotation value?
There was a problem hiding this comment.
For serverless/knative mode just passing the min max annotation under the spec.predictor.annotations is sufficient.
Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>
| SchedulerName string `json:"schedulerName,omitempty"` | ||
| Metrics Metrics `json:"metrics,omitempty"` | ||
| // +kubebuilder:validation:Minimum=1 | ||
| // +kubebuilder:validation:Minimum=0 |
There was a problem hiding this comment.
Okay I remember now. We had already added support to avoid querying the model details if the deployment is scaled down. This change makes sense with that context.
| if n.IsAutoScalingEnabled() { | ||
| return n.Spec.Scale.HPA.MinReplicas | ||
| } | ||
| return n.Spec.Replicas |
There was a problem hiding this comment.
What's the expected behavior when replicas is nil here? For deployments, if replicas is nil, it defaults to 1.
GROUP: apps
KIND: Deployment
VERSION: v1
FIELD: replicas <integer>
DESCRIPTION:
Number of desired pods. This is a pointer to distinguish between explicit
zero and not specified. Defaults to 1.
I'm assuming we want to keep this behavior here as well
There was a problem hiding this comment.
The replica on NIMservice can stay nil the corresponding deployment with will default to 1.
| if !n.IsAutoScalingEnabled() { | ||
| params.Replicas = n.GetReplicas() | ||
| } |
There was a problem hiding this comment.
| params.Replicas = n.GetReplicas() |
Its probably good practice to set this to HPA minReplicas if its enabled.
Signed-off-by: Vishesh Tanksale <vtanksale@nvidia.com>
3 participants
