Skip to content

Commit 9d5f1c2

Browse files
committed
[SPARK-57706] Add one-zone Spark Connect Server example
### What changes were proposed in this pull request? This PR aims to add one-zone `Spark Connect Server` example. ### Why are the changes needed? One-zone deployment is recommended to reduce the network cost among pods. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ``` $ kubectl get pod -l spark.operator/spark-app-name=spark-connect-server-usw2-az2 NAME READY STATUS RESTARTS AGE spark-connect-server-exec-1 1/1 Running 0 87s spark-connect-server-exec-2 1/1 Running 0 87s spark-connect-server-exec-3 1/1 Running 0 87s spark-connect-server-usw2-az2-0-driver 1/1 Running 0 93s ``` ``` $ kubectl get pod -l spark.operator/spark-app-name=spark-connect-server-usw2-az2 -oyaml | grep zone-id topology.k8s.aws/zone-id: usw2-az2 topology.k8s.aws/zone-id: usw2-az2 topology.k8s.aws/zone-id: usw2-az2 topology.k8s.aws/zone-id: usw2-az2 ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #718 from dongjoon-hyun/SPARK-57706. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 8813726 commit 9d5f1c2

1 file changed

Lines changed: 44 additions & 0 deletions

File tree

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
apiVersion: spark.apache.org/v1
16+
kind: SparkApplication
17+
metadata:
18+
name: spark-connect-server-usw2-az2
19+
spec:
20+
mainClass: "org.apache.spark.sql.connect.service.SparkConnectServer"
21+
sparkConf:
22+
spark.dynamicAllocation.enabled: "true"
23+
spark.dynamicAllocation.shuffleTracking.enabled: "true"
24+
spark.dynamicAllocation.minExecutors: "3"
25+
spark.dynamicAllocation.maxExecutors: "3"
26+
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
27+
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}"
28+
spark.kubernetes.executor.podNamePrefix: "spark-connect-server"
29+
spark.scheduler.mode: "FAIR"
30+
driverSpec:
31+
podTemplateSpec:
32+
spec:
33+
nodeSelector:
34+
topology.k8s.aws/zone-id: usw2-az2
35+
executorSpec:
36+
podTemplateSpec:
37+
spec:
38+
nodeSelector:
39+
topology.k8s.aws/zone-id: usw2-az2
40+
applicationTolerations:
41+
resourceRetainPolicy: OnFailure
42+
ttlAfterStopMillis: 10000
43+
runtimeVersions:
44+
sparkVersion: "4.1.2"

0 commit comments

Comments
 (0)