how superset worker pods are used and where to configure to utilise workloads in helm

[Nivraj]

how superset worker pods are used and where to configure to utilise workloads in helm

[dosubot[bot]]

Superset worker pods handle background tasks in Apache Superset, typically managed by a Celery worker. You can configure them in Helm under the supersetWorker section in the Helm chart's values.yaml file. This includes setting the number of replicas, resource limits, affinity, and autoscaling options. For example, you can enable autoscaling by setting supersetWorker.autoscaling.enabled to true and configure the minimum and maximum number of replicas with supersetWorker.autoscaling.minReplicas and supersetWorker.autoscaling.maxReplicas ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Nivraj]

how do i know weather its geting used.

[Nivraj]

@dosu do i need to configure any details., can you share complete set of configuration and code need to be used to enable celery worker

[dosubot[bot]]

To verify if the Superset worker pods are being utilized, you can check their status and logs using Kubernetes commands. For example, use kubectl get pods to see if the pods are running, and kubectl logs <pod-name> to view the logs and check if tasks are being processed.

To enable the Celery worker in Helm, you need to configure the values.yaml file with the following settings:

supersetWorker:
  replicas:
    enabled: true
    replicaCount: 1
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 100
    targetCPUUtilizationPercentage: 80
  podDisruptionBudget:
    enabled: false
    minAvailable: 1
    maxUnavailable: 1
  command:
    - "/bin/sh"
    - "-c"
    - ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app worker"
  forceReload: false
  initContainers:
    - name: wait-for-postgres-redis
      image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
      imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
      envFrom:
        - secretRef:
            name: "{{ tpl .Values.envFromSecret . }}"
      command:
        - /bin/sh
        - -c
        - dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT" -timeout 120s
  extraContainers: []
  deploymentAnnotations: {}
  deploymentLabels: {}
  affinity: {}
  livenessProbe:
    exec:
      command:
        - "celery"
        - "inspect"
        - "ping"
    initialDelaySeconds: 120
    periodSeconds: 60
    timeoutSeconds: 60
    successThreshold: 1
    failureThreshold: 3
  containerSecurityContext: {}
  podAnnotations: {}

This configuration sets up the Superset worker pods with a Celery worker command, initializes containers to wait for dependencies like PostgreSQL and Redis, and includes a liveness probe using the celery inspect ping command to ensure the worker is responsive ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Nivraj]

@dosu workers are running but im not seeing any logs in the pod., how do i verify if celery worker is geting utilised

[dosubot[bot]]

To verify if the Celery worker is being utilized, you can check the logs of the worker pod for any tasks being processed. If you're not seeing any logs, ensure that the SUPERSET_LOG_LEVEL environment variable is set correctly. By default, it is set to "info", but you can increase the verbosity to "debug" to capture more detailed logs. You can do this by modifying the values.yaml file in your Helm chart:

supersetWorker:
  environment:
    SUPERSET_LOG_LEVEL: "debug"

Additionally, you can troubleshoot missing logs by checking the health of the Celery worker using Kubernetes commands. Use kubectl logs <pod-name> to view the logs and kubectl describe pod <pod-name> to check for any issues with the pod's configuration or status. Ensure that the liveness probe is correctly configured to verify the worker's responsiveness:

livenessProbe:
  exec:
    command:
      - "celery"
      - "inspect"
      - "ping"
  initialDelaySeconds: 120
  periodSeconds: 60
  timeoutSeconds: 60
  successThreshold: 1
  failureThreshold: 3

If the health check fails, it might indicate issues with the Celery worker's connectivity or configuration ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}