Deploy Kyuubi on Kubernetes

[SimonYang-CS]

Hi Team,

I’m following the kyuubi document to deploy Kyuubi 1.4.0 on Kubernetes, , and I meet some issue I can’t resolve it.

Real appreciate if you could advise what is missing or what I do is wrong.

1. Preparation:
Run spark-submit with Spark-Pi on the target Kubernetes without any issue.

2. Configuration change in kyuubi-configmap.yaml :

kyuubi.authentication           NONE
kyuubi.frontend.bind.port       10009
kyuubi.ha.zookeeper.quorum      zookeepr-svc-name:2181
spark.master                    k8s:/<<api-url>
spark.submit.deployMode         client

3: What works:

kubectl port-forward svc/kyuubi-example-service -n <ns-here> 10009:10009

./beeline -u 'jdbc:hive2://localhost:10009/?spark.kubernetes.container.image=apache/kyuubi:master-snapshot;spark.kubernetes.namespace=<ns-here>;spark.kubernetes.authenticate.driver.serviceAccountName=spark;spark.dynamicAllocation.enabled=false;spark.shuffle.service.enabled=false'

0: jdbc:hive2://localhost:10009/> show databases;
+------------+
| namespace  |
+------------+
| default    |
+------------+

0: jdbc:hive2://localhost:10009/> show tables;
+-----------+------------+--------------+
| database  | tableName  | isTemporary  |
+-----------+------------+--------------+
+-----------+------------+--------------+
No rows selected (2.006 seconds)

4: What does not work as expected:

0: jdbc:hive2://localhost:10009/> select 1;
2021-12-22 10:53:13.952 INFO operation.ExecuteStatement: Query[116f759c-c25f-4356-9125-b1078c9d8211] in RUNNING_STATE
2021-12-22 10:53:18.954 INFO operation.ExecuteStatement: Query[116f759c-c25f-4356-9125-b1078c9d8211] in RUNNING_STATE

…..
RUNNING_STATE
RUNNING_STATE
RUNNING_STATE
RUNNING_STATE
RUNNING_STATE
RUNNING_STATE
…..

5. Log files in kyuubi engine(kyuubi-spark-sql-engine.log.0):

2021-12-22 10:54:35.336 WARN k8s.ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber.
org.apache.spark.SparkException: **Must specify the executor container image**
        at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$executorContainerImage$1(BasicExecutorFeatureStep.scala:44)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.<init>(BasicExecutorFeatureStep.scala:44)
        at org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBuilder.buildFromFeatures(KubernetesExecutorBuilder.scala:45)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$2(ExecutorPodsAllocator.scala:328)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:319)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$14(ExecutorPodsAllocator.scala:283)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$14$adapted(ExecutorPodsAllocator.scala:202)
        at scala.collection.Iterator.foreach(Iterator.scala:941)
        at scala.collection.Iterator.foreach$(Iterator.scala:941)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:202)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$1(ExecutorPodsAllocator.scala:103)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$1$adapted(ExecutorPodsAllocator.scala:103)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.org$apache$spark$scheduler$cluster$k8s$ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber$$processSnapshotsInternal(ExecutorPodsSnapshotsStoreImpl.scala:138)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.processSnapshots(ExecutorPodsSnapshotsStoreImpl.scala:126)
        at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl.$anonfun$addSubscriber$1(ExecutorPodsSnapshotsStoreImpl.scala:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

I try following setting without any luck:

spark.kubernetes.container.image=apache/kyuubi:master-snapshot;

Thanks for help

[yaooqinn]

spark.kubernetes.container.image=apache/kyuubi:master-snapshot

it shall be a spark docker image

[SimonYang-CS]

@yaooqinn , thanks for quick response.

./beeline -u 'jdbc:hive2://localhost:10009/?spark.kubernetes.container.image.pullSecrets=xx;spark.kubernetes.container.image=yy:v3.2.0...'

Caused by: java.io.IOException: Failed to connect to kyuubi-deployment-example-7fd4448b7b-7hph7:39041                                                                                                                                                                                     
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288)                                                                                                                                                                           
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)                                                                                                                                                                           
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)                                                                                                                                                                           
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)                                                                                                                                                                                                     
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)                                                                                                                                                                                                                    
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)                                                                                                                                                                                                                
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)                                                                                                                                                                                                                      
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                                                                        
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                                                                       
    at java.base/java.lang.Thread.run(Unknown Source)    

Caused by: java.net.UnknownHostException: kyuubi-deployment-example-7fd4448b7b-7hph7                                                                                                                                                                                                  
    at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source)                                                                                                                                                                                                             
    at java.base/java.net.InetAddress.getAllByName0(Unknown Source)                                                                                                                                                                                                                   
    at java.base/java.net.InetAddress.getAllByName(Unknown Source)                                                                                                                                                                                                                           
    at java.base/java.net.InetAddress.getAllByName(Unknown Source)                                                                                                                                                                                                                        
    at java.base/java.net.InetAddress.getByName(Unknown Source)  

Seems the spark executor can't connect to Kyuubi( spark Driver).

in a very short time, the driver created/deleted more than 400 executors.

kyuubiusersparksqlanonymousdefaulte1bbf73b-df32-41eb-8b75-e4abfe1c98f9-f4d63d7de21d1414-exec-405

[yaooqinn]

yes，currently we are fixing these issues in #1602 #1604

[SimonYang-CS]

@yaooqinn , we can change Kyuubi server to statefulset to resolve the UnknownHostException issue.

kyuubi@kyuubi-statefulset-example-2:/opt/kyuubi$ hostname
kyuubi-statefulset-example-2
kyuubi@kyuubi-statefulset-example-2:/opt/kyuubi$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.245.50.36    kyuubi-statefulset-example-2.kyuubi-server-headless.namespace.svc.cluster.local      kyuubi-statefulset-example-2

but I got a new issue:

java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef; local class incompatible: stream classdesc serialVersionUID = 1028182004549731694, local class serialVersionUID = 3456489343829468865  

I will try spark 3.1.2 instead of 3.2.0 and update here.

[SimonYang-CS]

@yaooqinn, thanks again for you help.

0: jdbc:hive2://localhost:10009/> select now() as date;
+--------------------------+
|           date           |
+--------------------------+
| 2021-12-23 02:33:57.339  |
+--------------------------+
1 row selected (3.534 seconds)

[yaooqinn]

Any time, enojoy~