K3S deploy rudder-server stdout only one message

[coyzeng]

Describe the bug
Deploy with k3s, rudder-server print only one message in stdout and exit.

2024/12/05 06:38:38 maxprocs: Updating GOMAXPROCS=5: determined from CPU quota

Steps to reproduce the bug

I'm not sure this bug can be reproduce with other environment. My runtime environment is K3S install on CentOS stream/Intel I7. Darwin can run rudder-server successfully.

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "13"
    meta.helm.sh/release-name: rudder
    meta.helm.sh/release-namespace: cis
  creationTimestamp: "2024-12-04T11:43:23Z"
  generation: 17
  labels:
    app.kubernetes.io/managed-by: Helm
    group: rudder
    set: rudder
  name: rudder
  namespace: cis
  resourceVersion: "45194742"
  uid: e9a8d86b-ed75-4d47-bf47-84941dd31248
spec:
  progressDeadlineSeconds: 60
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      group: rudder
      set: rudder
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        group: rudder
        set: rudder
    spec:
      containers:
        env:
        - name: JOBS_DB_HOST
          value: postgres.baas
        - name: JOBS_DB_PORT
          value: "5432"
        - name: JOBS_DB_USER
          value: postgres
        - name: JOBS_DB_PASSWORD
          value: xxx
        - name: JOBS_DB_DB_NAME
          value: rudder
        - name: DEST_TRANSFORM_URL
          value: http://transformer:9090
        - name: CONFIG_BACKEND_URL
        - name: WORKSPACE_TOKEN
          value: xxx
        - name: STATSD_SERVER_URL
          value: exporter.baas:9125
        - name: RSERVER_GATEWAY_WEBHOOK_SOURCE_LIST_FOR_PARSING_PARAMS
          value: Shopify
        image: rudderlabs/rudder-server:latest
        imagePullPolicy: IfNotPresent
        name: main
        resources:
          limits:
            cpu: "5"
            memory: 1G
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data
          subPath: data/rudder/data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 30
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: rudder

[contributor-support]

Thanks for opening this issue! We'll get back to you shortly. If it is a bug, please make sure to add steps to reproduce the issue.

[gitcommitshow]

Hi @coyzeng apologies, I missed this one. Were you able to figure this out?
Help me understand the issue and environment context in more depth. Did you verify the sever and transformer processes, whether they were running or not? What ram/cpu/disk resources available before and after running the server?

[jannikbend]

HI, we are every now and then running in the same error on our local development machines (using k3s as well). The issue is caused by not enough available memory on the host. Increasing the available memory for the docker vm or stopping other services then resolves the issue in our cases.

I think rudderstack does some kind of memory check and just exits if not enough memory is available. The next log line that should follow is: "INFO rudder-server/main.go:27 Setting memory limit to {"limit": 20126475878}".

[gitcommitshow]

@jannikbend understood. Please help me understand more about your runtime environment, I see you mentioned that you are using CentOS stream/Intel I7, and your deployment configuration sets limit on memory to be 1GB. What are the actual resources (CPU, RAM, etc.) available on the machine?

Edit: @coyzeng seeking your answer to above question, also do you have any additional update on the issue?

[jannikbend]

I'm not a colleague or anything of the one who originally raised the issue, I was just trying to help based on our experience with this particular bug.
We are using ARM MacBooks with 32 gigs of ram and ran into the issue when we only assigned below 16 gigs to the docker vm. From our experiences the issue is not k8s specific as well: The error can be reproduced with a simple "docker run rudderstack" as well (when there already are many services deployed).