Skip to content

Commit 9fbbf5e

Browse files
[helm] Expose user deployment replicaCount (#33908)
## Summary & Motivation I noticed that https://github.com/dagster-io/dagster/releases/tag/1.13.8 mentions multi-replica code locations for Dagster Cloud. Having multiple user deployment replicas with readiness probes enabled could improve stability and decrease occurrences of `DagsterUserCodeUnreachableError`. Dagster controller already connects to user deployment via k8s service, so this change is minimal. :) ## Test Plan Added tests to cover basic scenarios. ## Changelog > The changelog is generated by an agent that examines merged PRs and > summarizes/categorizes user-facing changes. You can optionally replace > this text with a terse description of any user-facing changes in your PR, > which the agent will prioritize. Otherwise, delete this section. Co-authored-by: Hynek Blaha <hynek.blaha@second-foundation.eu>
1 parent b3fe82a commit 9fbbf5e

7 files changed

Lines changed: 56 additions & 2 deletions

File tree

helm/dagster/charts/dagster-user-deployments/templates/deployment-user.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ metadata:
1313
{{- end }}
1414
annotations: {{ $deployment.annotations | toYaml | nindent 4 }}
1515
spec:
16-
replicas: 1
16+
replicas: {{ $deployment.replicaCount | default 1 }}
1717
{{- if $deployment.deploymentStrategy }}
1818
strategy:
1919
{{- toYaml $deployment.deploymentStrategy | nindent 4 }}

helm/dagster/charts/dagster-user-deployments/values.schema.json

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

helm/dagster/charts/dagster-user-deployments/values.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,9 @@ deployments:
5858
- "/example_project/example_repo/repo.py"
5959
port: 3030
6060

61+
# Number of replicas of the user code gRPC server Deployment.
62+
replicaCount: 1
63+
6164
# Whether or not to include configuration specified for this user code deployment in the pods
6265
# launched for runs from that deployment
6366
includeConfigInLaunchedRuns:

helm/dagster/schema/schema/charts/dagster_user_deployments/subschema/user_deployments.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from pydantic import BaseModel, create_model
1+
from pydantic import BaseModel, Field, create_model
22

33
from schema.charts.utils import kubernetes
44

@@ -20,6 +20,7 @@ class UserDeployment(BaseModel):
2020
includeConfigInLaunchedRuns: UserDeploymentIncludeConfigInLaunchedRuns | None = None
2121
deploymentNamespace: str | None = None
2222
port: int
23+
replicaCount: int = Field(default=1, gt=0)
2324
env: dict[str, str] | list[kubernetes.EnvVar] | None = None
2425
envConfigMaps: list[kubernetes.ConfigMapEnvSource] | None = None
2526
envSecrets: list[kubernetes.SecretEnvSource] | None = None

helm/dagster/schema/schema_tests/test_user_deployments.py

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -857,6 +857,41 @@ def _assert_has_container_context(user_deployment):
857857
assert "DAGSTER_CLI_API_GRPC_CONTAINER_CONTEXT" in env_names
858858

859859

860+
def test_user_deployment_replica_count_default_is_one(template: HelmTemplate):
861+
deployment = create_simple_user_deployment("foo")
862+
helm_values = DagsterHelmValues.construct(
863+
dagsterUserDeployments=UserDeployments.construct(deployments=[deployment])
864+
)
865+
866+
user_deployments = template.render(helm_values)
867+
assert len(user_deployments) == 1
868+
assert user_deployments[0].spec.replicas == 1
869+
870+
871+
@pytest.mark.parametrize("replica_count", [1, 2])
872+
def test_user_deployment_replica_count(template: HelmTemplate, replica_count: int):
873+
deployment = create_simple_user_deployment("foo")
874+
deployment.replicaCount = replica_count
875+
helm_values = DagsterHelmValues.construct(
876+
dagsterUserDeployments=UserDeployments.construct(deployments=[deployment])
877+
)
878+
879+
user_deployments = template.render(helm_values)
880+
assert len(user_deployments) == 1
881+
assert user_deployments[0].spec.replicas == replica_count
882+
883+
884+
def test_user_deployment_replica_count_rejects_zero():
885+
with pytest.raises(Exception):
886+
UserDeployment(
887+
name="foo",
888+
image=kubernetes.Image(repository="repo/foo", tag="tag1", pullPolicy="Always"),
889+
dagsterApiGrpcArgs=["-m", "foo"],
890+
port=3030,
891+
replicaCount=0,
892+
)
893+
894+
860895
def test_user_deployment_image(template: HelmTemplate):
861896
deployment = create_simple_user_deployment("foo")
862897
helm_values = DagsterHelmValues.construct(

helm/dagster/values.schema.json

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

helm/dagster/values.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,9 @@ dagster-user-deployments:
365365
- "/example_project/example_repo/repo.py"
366366
port: 3030
367367

368+
# Number of replicas of the user code gRPC server Deployment.
369+
replicaCount: 1
370+
368371
# Whether or not to include configuration specified for this user code deployment in the pods
369372
# launched for runs from that deployment
370373
includeConfigInLaunchedRuns:

0 commit comments

Comments
 (0)