zeebe and zeebe-gateways writes logs under the container's filesystem even `readOnlyRootFilesystem` is set to `true`

[Dentrax]

Describe the bug

zeebe-gateway writes logs to its read-only filesystem even securityContext. readOnlyRootFilesystem is set to true.

To Reproduce

Spin up a new camunda platform using the Kubernetes Helm Chart.

Expected behavior

Container should be fully read-only. Even $ZEEBE_LOG_APPENDER defaults to Console, all zeebe and zeebe-gateway instances writes logs under /usr/local/zeebe/logs.

Log/Stacktrace

camunda@camunda-zeebe-zeebe-gateway-75b4fcb5d7-448wx:~$ ls /usr/local/zeebe/logs/
zeebe.log
camunda@camunda-zeebe-zeebe-gateway-75b4fcb5d7-448wx:~$ cat /usr/local/zeebe/logs/zeebe.log 
2024-08-27 16:03:23.267 [] [main] [] INFO 
      io.camunda.zeebe.gateway.StandaloneGateway - Starting StandaloneGateway v8.5.6 using Java 21.0.4 with PID 6 (/usr/local/zeebe/lib/camunda-zeebe-8.5.6.jar started by camunda in /usr/local/zeebe)
...
...
...

Full Stacktrace

<STACKTRACE>

Environment:

• OS: Kubernetes
• Zeebe Version: camunda/zeebe:8.5.6
• Configuration: Helm Chart default

[npepinpe]

Normally /usr/camunda/zeebe/logs should be mounted as a volume, even if it's only EmptyDir (for non-persistent logs). To be honest, in the cloud, I would fully disable the file logging. It looks like this is not currently possible without overwriting the default configuration file. I'll open an FR for it, in the mean time I will redirect this PR to the Helm chart repository.

[falko]

My current prospect ran into the same problem. @npepinpe did you get a chance to draft a PR or did you pass it on to @aabouzaid?

[falko]

Is there any workaround, e.g. turning off the file appender maybe using the zeebe.log4j chart value?

[npepinpe]

With the latest patches, this is easily doable via environment variables, and documented. Available for 8.6.12 (or 8.6.13?).

In the meantime, anyone can add their own log4j2.xml and remove the rolling file appender, basically.

[falko]

@npepinpe could you share a link to that documentation or the name & value of the environment variable?