Slower than Spark UI in Kubernetes

[premkumar-masilamani]

Hi - Thank you for the spark-ui-proxy. It made my life much easier. I no longer have to port-forward to each of the worker nodes.

I used this proxy with spark version 1.6.3 and deployed as pods (master, worker, ui-proxy each in separate pods) in kubernetes cluster. Here is the deployment file. The proxy is slower than I expected. It takes around 2 - 3 seconds.

Direct port-forwarding to the master/worker nodes are super fast.

Is there any specific resource requirements for this proxy? CPU, RAM etc?. Anything we could do to speed up?

---
apiVersion: v1
kind: Service
metadata:
  name: spark-ui-proxy
  labels:
    component: spark-ui-proxy
spec:
  ports:
    - port: 80
      targetPort: 80
  clusterIP: None
  selector:
    component: spark-ui-proxy
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: spark-ui-proxy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        component: spark-ui-proxy
    spec:
      containers:
        - name: spark-ui-proxy
          image: networkaispark/spark-ui-proxy:1.0.0
          ports:
            - containerPort: 80
          args:
            - spark-master-v1:8080
          livenessProbe:
              httpGet:
                path: /
                port: 80
              initialDelaySeconds: 120
              timeoutSeconds: 5

[aseigneurin]

I never tested with Kubernetes. I don't expect the proxy to use a high amount of CPU or memory.

Did you find a solution to this problem?