Skip to content

spark-ui-proxy for spark standalone cluster with spark master configured in HA #22

@kiransgithub

Description

@kiransgithub

Thanks for creating this spark-ui-proxy. In my case, I have a spark standalone cluster with spark master configured with zookeeper to achieve spark master high availability. There are couple of nodes running spark master service in ALIVE (Active) and STANDBY mode.
I had to run this spark-ui-proxy on both the master instances. When I use a apache httpd virtual hosting for load balancing this, requests are not sticking to ALIVE spark master, instead it is going in round robin fashion as both the spark-ui-proxy servers are listen on a port and responding with a status!

To overcome this, I am trying to depend on mod_proxy_hcheck of apache httpd which will disable the balancer member based on the GET calls response. Unfortunately , with this spark-ui-proxy I am getting OK status on both the spark master which again takes me back to round-robin issue between ALIVE and STANDBY master nodes.

Within your code , I see below for health checking, which is returning OK in both the cases:

Add an health checking endpoint.

    if self.path in ["/healthz"]:
        self.send_response(code=200)
        self.send_header("Content-type", "text/plain")
        self.end_headers()
        self.wfile.write("OK")
        return

Is it possible to return ACTIVE or STANDBY responses which will really help in achiving HA with spark-master using this spark-ui-proxy ?

Appreciate any help.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions