Question: Does Sail have a deployment mode similar to Spark StandAlone?

[xunxunmimi5577]

Can I manually start the worker and register with the driver instead of using k8s?

[linhr]

Thanks for reaching out!

Sail doesn't support standalone mode yet. Here are a few considerations if we'd like to explore this idea.

1. Right now, the driver for each session listens on an arbitrary port and propagates the port information to the worker when launching the worker. To support manual worker registration, we need the driver port to be a configurable static port so that the worker can connect to the driver. The challenge is that the Sail server can run multiple sessions each with an isolated driver, so we need to figure out a story of assigning ports for drivers.
2. Right now, the driver autoscales the worker depending on the workload and removes the worker when it is no longer in use. If I understand correctly, the standalone mode operates under a fixed capacity so we need to skip the auto-scaling logic in the driver.

We would love to hear your use cases! The standalone mode with manual worker registration is not a common production setting that we're aware of, but let us know your requirements and we'd be happy to see how we can better support them.

(BTW, Sail has a local cluster mode where each driver/worker runs in separate threads within the same Sail server process. This is useful for testing purposes.)

[xunxunmimi5577]

Thank s for your answer.Another question, why doesn't sail delete the completed pods?

[linhr]

why doesn't sail delete the completed pods?

The worker pods will be removed after the server pod terminates. This allows the user to have a way to access worker pod logs if needed. The Sail server is designed to be lightweight instead of long-running, so it is typical for users to start a new server for each data processing job and all the pods (workers and server) will be cleaned up once the job finishes.

I can see that automatic worker cleanup might be desirable if the server is long-running. This usage pattern is somewhat different from Sail's design philosophy, but we'd love to learn about the use cases!

[xunxunmimi5577]

why doesn't sail delete the completed pods?

The worker pods will be removed after the server pod terminates. This allows the user to have a way to access worker pod logs if needed. The Sail server is designed to be lightweight instead of long-running, so it is typical for users to start a new server for each data processing job and all the pods (workers and server) will be cleaned up once the job finishes.

I can see that automatic worker cleanup might be desirable if the server is long-running. This usage pattern is somewhat different from Sail's design philosophy, but we'd love to learn about the use cases!

I saw this description in your document, “The Sail server in cluster mode can support multiple sessions, each powered by a cluster job runner. Each cluster job runner owns a driver that schedules the distributed physical plan containing multiple stages.” So I thought Sail Server was a long-running service.

[linhr]

Yeah the Spark Connect protocol supports the concept of a "session", so multiple clients can connect to the same server. But in production settings, it is usually good to have a separate Spark Connect server for each job for better resource isolation.

[xunxunmimi5577]

Hi~ @linhr, thanks for your time again. Does Sail support partitioned tables?The Memory catalog doesn't seem to support it.I'm not sure if other catalogs support it?

[linhr]

Yes Sail supports partitioned table. If memory catalog or other catalog have problems with it, that needs to be fixed. Is the problem you encountered related to the PARTITIONED BY SQL clause or directory layout for Hive-style partitioning? Feel free to create a dedicated issue with examples and we'll take a look!

[xunxunmimi5577]

Yes Sail supports partitioned table. If memory catalog or other catalog have problems with it, that needs to be fixed. Is the problem you encountered related to the PARTITIONED BY SQL clause or directory layout for Hive-style partitioning? Feel free to create a dedicated issue with examples and we'll take a look!

Using memory catalog and pyspark client mod, I can create a partitioned table.

spark.sql("create table test_part(id int) using parquet partitioned by (part string)").show();

then I select from the table,there will be an exception as follows.

spark-3.5.8-bin-hadoop3/python/pyspark/sql/connect/client/core.py", line 1503, in _handle_error
self._handle_rpc_error(error)
File "/cloud/service/spark-3.5.8-bin-hadoop3/python/pyspark/sql/connect/client/core.py", line 1539, in _handle_rpc_error
raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.AnalysisException: missing or ambiguous partition column: part

And if I describe the table , I can't see the partition column.
spark.sql("describe table test_part").show()
+--------+---------+-------+
|col_name|data_type|comment|
+--------+---------+-------+
| id| int| NULL|
+--------+---------+-------+

[xunxunmimi5577]

Yes Sail supports partitioned table. If memory catalog or other catalog have problems with it, that needs to be fixed. Is the problem you encountered related to the PARTITIONED BY SQL clause or directory layout for Hive-style partitioning? Feel free to create a dedicated issue with examples and we'll take a look!

Using memory catalog and pyspark client mod, I can create a partitioned table.

spark.sql("create table test_part(id int) using parquet partitioned by (part string)").show();

then I select from the table,there will be an exception as follows.

spark-3.5.8-bin-hadoop3/python/pyspark/sql/connect/client/core.py", line 1503, in _handle_error self._handle_rpc_error(error) File "/cloud/service/spark-3.5.8-bin-hadoop3/python/pyspark/sql/connect/client/core.py", line 1539, in _handle_rpc_error raise convert_exception(info, status.message) from None pyspark.errors.exceptions.connect.AnalysisException: missing or ambiguous partition column: part

And if I describe the table , I can't see the partition column. spark.sql("describe table test_part").show() +--------+---------+-------+ |col_name|data_type|comment| +--------+---------+-------+ | id| int| NULL| +--------+---------+-------+

I understand now, it seems that Sail does not support Hive style PARTIONED BY syntax?

[linhr]

Thanks for the example! This seems an issue when the partition column is not defined in the column list. I created #1588 to track this.