Hi, I was digging through the SparkClient code and noticed something.
Spark itself comes with a built in dashboard that shows you what a job is doing while it runs and after it finishes, which steps are done, which one is stuck, how long each part took. It also supports sending out Prometheus metrics through a sidecar. Both of these are just fields on the SparkApplication spec, sparkUIOptions and monitoring, that the operator already understands and supports.
I checked both places where SparkClient builds a SparkApplication, get_spark_application_cr_from_file_job and get_spark_application_cr_from_func_job in kubeflow/spark/backends/kubernetes/utils.py, and neither field ever gets set. I also checked the public SparkClient API in spark_client.py and there is nothing there about the UI or metrics either.
So right now if you submit a job with submit_job() and it hangs or runs slow, the only thing you can look at is raw logs through get_job_logs(). There is no way to actually see the job, even though Spark already has that dashboard built and ready, this SDK just never connects it.
I would like to work on this. My plan is to add an option to submit_job() that turns on the UI (something like enable_ui) and wires up spark_ui_options with a sensible default, plus a small helper like get_job_ui_url() that reuses the same port forward approach already used for Spark Connect in get_connect_url(). I think monitoring can be a separate follow up since it is really its own feature (the Prometheus sidecar) and not just a missing wire.
Let me know if this sounds like the right direction before I start on a PR.
Hi, I was digging through the SparkClient code and noticed something.
Spark itself comes with a built in dashboard that shows you what a job is doing while it runs and after it finishes, which steps are done, which one is stuck, how long each part took. It also supports sending out Prometheus metrics through a sidecar. Both of these are just fields on the SparkApplication spec,
sparkUIOptionsandmonitoring, that the operator already understands and supports.I checked both places where SparkClient builds a SparkApplication,
get_spark_application_cr_from_file_jobandget_spark_application_cr_from_func_jobinkubeflow/spark/backends/kubernetes/utils.py, and neither field ever gets set. I also checked the public SparkClient API inspark_client.pyand there is nothing there about the UI or metrics either.So right now if you submit a job with
submit_job()and it hangs or runs slow, the only thing you can look at is raw logs throughget_job_logs(). There is no way to actually see the job, even though Spark already has that dashboard built and ready, this SDK just never connects it.I would like to work on this. My plan is to add an option to
submit_job()that turns on the UI (something likeenable_ui) and wires upspark_ui_optionswith a sensible default, plus a small helper likeget_job_ui_url()that reuses the same port forward approach already used for Spark Connect inget_connect_url(). I think monitoring can be a separate follow up since it is really its own feature (the Prometheus sidecar) and not just a missing wire.Let me know if this sounds like the right direction before I start on a PR.