[ISSUE] Zeebe exporters dir mounted as read-only

[rohits123]

Describe the issue:

I am adding custom kafka exporter the "Adding dynamic exporters to Zeebe Broker" - way, using Helm charts (version 8.3.3) into Kubernetes.
Zeebe config in values file

zeebe:
  pvcAccessModes:
  - ReadWriteMany
  pvcStorageClassName: "efs-sc"
  env:
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_CLASSNAME
    value: au.th.zeebe.boundary.CamundaExporter
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_JARPATH
    value: /exporters/zeebe-kafka-exporter.jar
  initContainers:
  - name: init-exporters-kafka
    image: amazon/aws-cli:2.15.8
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
    command: ["sh", "-c", "aws s3 cp s3://my-bucket/polaris-zeebe-exporter-1.0.0.jar /exporters/zeebe-kafka-exporter.jar && ls -al /exporters"]
    volumeMounts:
    - name: exporters
      mountPath: /exporters/
    env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: access-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: secret-access-key   

Actual behavior:

mkdir: cannot create directory ‘/usr/local/zeebe/exporters/’: Read-only file system     

Expected behavior:

The custom kafka exporter integration should work

How to reproduce:

Logs:

zeebe pod logs:

│ + export ZEEBE_BROKER_CLUSTER_NODEID=2                                                                                                                                                              │
│ + ZEEBE_BROKER_CLUSTER_NODEID=2                                                                                                                                                                     │
│ ++ ls -A /exporters/                                                                                                                                                                                │
│ + '[' zeebe-kafka-exporter.jar ']'                                                                                                                                                                  │
│ + mkdir -p /usr/local/zeebe/exporters/                                                                                                                                                              │
│ mkdir: cannot create directory ‘/usr/local/zeebe/exporters/’: Read-only file system                                                                                                                 │
│ Stream closed EOF for bpm/camunda-platform-dev-zeebe-2 (zeebe)    

initcontainer logs:

│ Completed 6.2 MiB/14.0 MiB (18.3 MiB/s) with 1 file(s) remaining Completed 6.5 MiB/14.0 MiB (19.0 MiB/s) with 1 file(s) remaining Completed 6.8 MiB/14.0 MiB (18.1 MiB/s) with 1 file(s) remaining  │
│ Completed 7.0 MiB/14.0 MiB (18.7 MiB/s) with 1 file(s) remaining Completed 7.2 MiB/14.0 MiB (19.3 MiB/s) with 1 file(s) remaining Completed 7.5 MiB/14.0 MiB (20.0 MiB/s) with 1 file(s) remaining  │
│ Completed 7.8 MiB/14.0 MiB (20.6 MiB/s) with 1 file(s) remaining Completed 8.0 MiB/14.0 MiB (21.2 MiB/s) with 1 file(s) remaining Completed 8.2 MiB/14.0 MiB (21.8 MiB/s) with 1 file(s) remaining  │
│ Completed 8.5 MiB/14.0 MiB (22.4 MiB/s) with 1 file(s) remaining Completed 8.8 MiB/14.0 MiB (23.0 MiB/s) with 1 file(s) remaining Completed 9.0 MiB/14.0 MiB (23.5 MiB/s) with 1 file(s) remaining  │
│ Completed 9.2 MiB/14.0 MiB (24.1 MiB/s) with 1 file(s) remaining Completed 9.5 MiB/14.0 MiB (24.7 MiB/s) with 1 file(s) remaining Completed 9.8 MiB/14.0 MiB (25.2 MiB/s) with 1 file(s) remaining  │
│ Completed 10.0 MiB/14.0 MiB (22.6 MiB/s) with 1 file(s) remainingCompleted 10.2 MiB/14.0 MiB (23.1 MiB/s) with 1 file(s) remainingCompleted 10.5 MiB/14.0 MiB (23.6 MiB/s) with 1 file(s) remaining │
│ Completed 10.8 MiB/14.0 MiB (24.1 MiB/s) with 1 file(s) remainingCompleted 11.0 MiB/14.0 MiB (24.6 MiB/s) with 1 file(s) remainingCompleted 11.2 MiB/14.0 MiB (25.2 MiB/s) with 1 file(s) remaining │
│ Completed 11.5 MiB/14.0 MiB (25.7 MiB/s) with 1 file(s) remainingCompleted 11.8 MiB/14.0 MiB (26.2 MiB/s) with 1 file(s) remainingCompleted 12.0 MiB/14.0 MiB (26.7 MiB/s) with 1 file(s) remaining │
│ Completed 12.2 MiB/14.0 MiB (22.5 MiB/s) with 1 file(s) remainingCompleted 12.5 MiB/14.0 MiB (22.9 MiB/s) with 1 file(s) remainingCompleted 12.8 MiB/14.0 MiB (23.4 MiB/s) with 1 file(s) remaining │
│ Completed 13.0 MiB/14.0 MiB (23.8 MiB/s) with 1 file(s) remainingCompleted 13.2 MiB/14.0 MiB (24.2 MiB/s) with 1 file(s) remainingCompleted 13.5 MiB/14.0 MiB (24.7 MiB/s) with 1 file(s) remaining │
│ Completed 13.8 MiB/14.0 MiB (25.1 MiB/s) with 1 file(s) remainingCompleted 14.0 MiB/14.0 MiB (25.5 MiB/s) with 1 file(s) remainingCompleted 14.0 MiB/14.0 MiB (25.6 MiB/s) with 1 file(s) remaining │
│ download: s3://my-bucket/polaris-zeebe-exporter-1.0.0.jar to ../exporters/zeebe-kafka-exporter.jar                                                              │
│ total 14384                                                                                                                                                                                         │
│ drwxrwsrwx 2 root 1000       38 Jan 10 23:35 .                                                                                                                                                      │
│ drwxr-xr-x 1 root root       45 Jan 10 23:35 ..                                                                                                                                                     │
│ -rw-r--r-- 1 1000 1000 14726752 Jan 10 00:59 zeebe-kafka-exporter.jar                                                                                                                               │
│ Stream closed EOF for bpm/camunda-platform-dev-zeebe-2 (init-exporters-kafka)  

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

• Platform: AWS, EKS
• Helm CLI version: 3.12.3
• Chart version: 8.3.3
• Values file:

[rohits123]

Update:

I have tried a separate mount path like below:

zeebe:
  pvcAccessModes:
  - ReadWriteMany
  pvcStorageClassName: "efs-sc"
  env:
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_CLASSNAME
    value: au.th.zeebe.boundary.CamundaExporter
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_JARPATH
    value: /tmp/zeebe-kafka-exporter.jar
  initContainers:
  - name: init-exporters-kafka
    image: amazon/aws-cli:2.15.8
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
    command: ["sh", "-c", "ls -ltr  /tmp && aws s3 cp s3://my-bucket/polaris-zeebe-exporter-1.0.0.jar /tmp/zeebe-kafka-exporter.jar && ls -al /tmp"]
    volumeMounts:
    - name: exporters-zeebe
      mountPath: /tmp/
    env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: access-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: secret-access-key   

  extraVolumes:
  - name: exporters-zeebe
    emptyDir: {}
  extraVolumeMounts:
  - name: exporters-zeebe
    mountPath: /tmp

But it fails to load exporter

+ export ZEEBE_BROKER_CLUSTER_NODEID=2                                                                                                                                                                        │
│ + ZEEBE_BROKER_CLUSTER_NODEID=2                                                                                                                                                                               │
│ ++ ls -A /exporters/                                                                                                                                                                                          │
│ + '[' '' ']'                                                                                                                                                                                                  │
│ + echo 'No exporters available.'                                                                                                                                                                              │
│ + exec /usr/local/zeebe/bin/broker                                                                                                                                                                            │
│ No exporters available.                                                                                                                                                                                       │
│ Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError                         │
│   ______  ______   ______   ____    ______     ____    _____     ____    _  __  ______   _____                                                                                                                │
│  |___  / |  ____| |  ____| |  _ \  |  ____|   |  _ \  |  __ \   / __ \  | |/ / |  ____| |  __ \                                                                                                               │
│     / /  | |__    | |__    | |_) | | |__      | |_) | | |__) | | |  | | | ' /  | |__    | |__) |                                                                                                              │
│    / /   |  __|   |  __|   |  _ <  |  __|     |  _ <  |  _  /  | |  | | |  <   |  __|   |  _  /                                                                                                               │
│   / /__  | |____  | |____  | |_) | | |____    | |_) | | | \ \  | |__| | | . \  | |____  | | \ \                                                                                                               │
│  /_____| |______| |______| |____/  |______|   |____/  |_|  \_\  \____/  |_|\_\ |______| |_|  \_\                                                                                                              │
│                                                                                                                                                                                                               │
│ 2024-01-12 05:46:14.070 [] [main] [] INFO                                                                                                                                                                     │
│       io.camunda.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v8.3.3 using Java 17.0.9 with PID 7 (/usr/local/zeebe/lib/camunda-zeebe-8.3.3.jar started by zeebe in /usr/local/zeebe)            │
│ 2024-01-12 05:46:14.074 [] [main] [] INFO                                                                                                                                                                     │
│       io.camunda.zeebe.broker.StandaloneBroker - The following 1 profile is active: "broker"                                                                                                                  │
│ 2024-01-12 05:46:18.803 [] [main] [] INFO                                                                                                                                                                     │
│       org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 10 endpoint(s) beneath base path '/actuator'                                                                             │
│ 2024-01-12 05:46:19.668 [] [main] [] INFO                                                                                                                                                                     │
│       org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port 9600                                                                                                                 │
│ 2024-01-12 05:46:19.682 [] [main] [] INFO                                                                                                                                                                     │
│       io.camunda.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 6.711 seconds (process running for 8.973)                                                                                        │
│ 2024-01-12 05:46:19.687 [] [main] [] WARN                                                                                                                                                                     │
│       io.camunda.zeebe.broker.system - No backup store is configured. Backups will not be taken                                                                                                               │
│ 2024-01-12 05:46:19.886 [] [main] [] INFO                                                                                                                                                                     │
│       org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLogger -                                                                                                                        │
│                                                                                                                                                                                                               │
│ Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.                                                                                   │
│ 2024-01-12 05:46:19.959 [] [main] [] ERROR                                                                                                                                                                    │
│       org.springframework.boot.SpringApplication - Application run failed                                                                                                                                     │
│ java.lang.IllegalStateException: Failed to execute CommandLineRunner                                                                                                                                          │
│     at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:774) ~[spring-boot-3.1.5.jar:3.1.5]                                                                                       │
│     at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:755) [spring-boot-3.1.5.jar:3.1.5]                                                                              │
│     at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?]                                                                                                                              │
│     at java.util.stream.SortedOps$SizedRefSortingSink.end(Unknown Source) ~[?:?]                                                                                                                              │
│     at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]                                                                                                                                      │
│     at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]                                                                                                                               │
│     at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]                                                                                                                        │
│     at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]                                                                                                                  │
│     at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]                                                                                                                                      │
│     at java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?]                                                                                                                                      │
│     at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:750) [spring-boot-3.1.5.jar:3.1.5]                                                                                       │
│     at org.springframework.boot.SpringApplication.run(SpringApplication.java:323) [spring-boot-3.1.5.jar:3.1.5]                                                                                               │
│     at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:78) [camunda-zeebe-8.3.3.jar:8.3.3]                                                                                                │
│ Caused by: java.lang.IllegalStateException: Failed to load exporter with configuration: ExporterCfg{, jarPath='/tmp/zeebe-kafka-exporter.jar', className='au.th.zeebe.boundary.CamundaExporter', args=null}
Caused by: java.lang.IllegalStateException: Failed to load exporter with configuration: ExporterCfg{, jarPath='/tmp/zeebe-kafka-exporter.jar', className='au.th.zeebe.boundary.CamundaExporter', args=null}   │
│     at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:149) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                │
│     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:70) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                                  │
│     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:49) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                                  │
│     at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                                                │
│     at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.5.jar:3.1.5]                                                                                       │
│     ... 12 more                                                                                                                                                                                               │
│ Caused by: io.camunda.zeebe.util.jar.ExternalJarLoadException: Cannot load JAR at [/tmp/zeebe-kafka-exporter.jar]: is not readable                                                                            │
│     at io.camunda.zeebe.util.jar.ExternalJarRepository.verifyJarPath(ExternalJarRepository.java:78) ~[zeebe-util-8.3.3.jar:8.3.3]                                                                             │
│     at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:54) ~[zeebe-util-8.3.3.jar:8.3.3]                                                                                      │
│     at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:47) ~[zeebe-util-8.3.3.jar:8.3.3]                                                                                      │
│     at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.load(ExporterRepository.java:75) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                              │
│     at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:147) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                │
│     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:70) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                                  │
│     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:49) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                                  │
│     at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                                                │
│     at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.5.jar:3.1.5]                                                                                       │
│     ... 12 more                                                                                                                                                                                               │
│ 2024-01-12 05:46:19.969 [] [main] [] WARN                                                                                                                                                                     │
│       org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent                               │
│ java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.broker.Broker.close()" because "this.broker" is null                                                                                          │
│     at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:94) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                                 │
│     at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:34) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                                 │
│     at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:174) ~[spring-context-6.0.13.jar:6.0.13]                                   │
│     at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:167) ~[spring-context-6.0.13.jar:6.0.13]                                     │
│     at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) ~[spring-context-6.0.13.jar:6.0.13]                                     │
│     at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:445) ~[spring-context-6.0.13.jar:6.0.13]                                                   │
│     at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) ~[spring-context-6.0.13.jar:6.0.13]                                                   │
│     at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1067) [spring-context-6.0.13.jar:6.0.13]                                                        │
│     at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:149) [spring-boot-3.1.5.jar:3.1.5]                                  │
│     at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1035) [spring-context-6.0.13.jar:6.0.13]                                                          │
│     at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:790) [spring-boot-3.1.5.jar:3.1.5]                                                                                  │
│     at org.springframework.boot.SpringApplication.run(SpringApplication.java:329) [spring-boot-3.1.5.jar:3.1.5]                                                                                               │
│     at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:78) [camunda-zeebe-8.3.3.jar:8.3.3]                                                                                                │
│ 2024-01-12 05:46:19.971 [] [main] [] INFO                                                                                                                                                                     │
│       org.springframework.boot.web.embedded.netty.GracefulShutdown - Commencing graceful shutdown. Waiting for active requests to complete                                                                    │
│ 2024-01-12 05:46:19.973 [] [netty-shutdown] [] INFO                                                                                                                                                           │
│       org.springframework.boot.web.embedded.netty.GracefulShutdown - Graceful shutdown complete                                                                                                               │
│ 2024-01-12 05:46:20.047 [] [atomix-cluster-0] [] INFO                                                                                                                                                         │
│       io.atomix.cluster.AtomixCluster - Stopped 

[aabouzaid]

Hi @rohits123 👋
You need to make the exporter path writable

zeebe:
[...]
  extraVolumes
  - name: exporters-zeebe
    emptyDir: {}
[...]
  extraVolumeMounts:
  - name: exporters-zeebe
    mountPath: /usr/local/zeebe/exporters

[rohits123]

Hey Ahmed, after making the path writable I get "Cannot load JAR at [/exporters/zeebe-kafka-exporter.jar]: is not readable"

zeebe + export ZEEBE_BROKER_CLUSTER_NODEID=2                                                                                                                                                                  │
│ zeebe + ZEEBE_BROKER_CLUSTER_NODEID=2                                                                                                                                                                         │
│ zeebe ++ ls -A /exporters/                                                                                                                                                                                    │
│ zeebe + '[' '' ']'                                                                                                                                                                                            │
│ zeebe + echo 'No exporters available.'                                                                                                                                                                        │
│ zeebe + exec /usr/local/zeebe/bin/broker                                                                                                                                                                      │
│ zeebe No exporters available.                                                                                                                                                                                 │
│ zeebe Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError                   │
│ zeebe   ______  ______   ______   ____    ______     ____    _____     ____    _  __  ______   _____                                                                                                          │
│ zeebe  |___  / |  ____| |  ____| |  _ \  |  ____|   |  _ \  |  __ \   / __ \  | |/ / |  ____| |  __ \                                                                                                         │
│ zeebe     / /  | |__    | |__    | |_) | | |__      | |_) | | |__) | | |  | | | ' /  | |__    | |__) |                                                                                                        │
│ zeebe    / /   |  __|   |  __|   |  _ <  |  __|     |  _ <  |  _  /  | |  | | |  <   |  __|   |  _  /                                                                                                         │
│ zeebe   / /__  | |____  | |____  | |_) | | |____    | |_) | | | \ \  | |__| | | . \  | |____  | | \ \                                                                                                         │
│ zeebe  /_____| |______| |______| |____/  |______|   |____/  |_|  \_\  \____/  |_|\_\ |______| |_|  \_\                                                                                                        │
│ zeebe                                                                                                                                                                                                         │
│ zeebe 2024-01-12 23:28:52.771 [] [main] [] INFO                                                                                                                                                               │
│ zeebe       io.camunda.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v8.3.3 using Java 17.0.9 with PID 8 (/usr/local/zeebe/lib/camunda-zeebe-8.3.3.jar started by zeebe in /usr/local/zeebe)      │
│ zeebe 2024-01-12 23:28:52.775 [] [main] [] INFO                                                                                                                                                               │
│ zeebe       io.camunda.zeebe.broker.StandaloneBroker - The following 1 profile is active: "broker"                                                                                                            │
│ zeebe 2024-01-12 23:28:57.101 [] [main] [] INFO                                                                                                                                                               │
│ zeebe       org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 10 endpoint(s) beneath base path '/actuator'                                                                       │
│ zeebe 2024-01-12 23:28:57.983 [] [main] [] INFO                                                                                                                                                               │
│ zeebe       org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port 9600                                                                                                           │
│ zeebe 2024-01-12 23:28:58.051 [] [main] [] INFO                                                                                                                                                               │
│ zeebe       io.camunda.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 6.375 seconds (process running for 8.634)                                                                                  │
│ zeebe 2024-01-12 23:28:58.058 [] [main] [] WARN                                                                                                                                                               │
│ zeebe       io.camunda.zeebe.broker.system - No backup store is configured. Backups will not be taken                                                                                                         │
│ zeebe 2024-01-12 23:28:58.250 [] [main] [] INFO                                                                                                                                                               │
│ zeebe       org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLogger -                                                                                                                  │
│ zeebe                                                                                                                                                                                                         │
│ zeebe Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.                                                                             │
│ zeebe 2024-01-12 23:28:58.278 [] [main] [] ERROR                                                                                                                                                              │
│ zeebe       org.springframework.boot.SpringApplication - Application run failed                                                                                                                               │
│ zeebe java.lang.IllegalStateException: Failed to execute CommandLineRunner                                                                                                                                    │
│ zeebe     at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:774) ~[spring-boot-3.1.5.jar:3.1.5]                                                                                 │
│ zeebe     at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:755) [spring-boot-3.1.5.jar:3.1.5]                                                                        │
│ zeebe     at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?]                                                                                                                        │
│ zeebe     at java.util.stream.SortedOps$SizedRefSortingSink.end(Unknown Source) ~[?:?]                                                                                                                        │
│ zeebe     at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]                                                                                                                                │
│ zeebe     at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]                                                                                                                         │
│ zeebe     at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]                                                                                                                  │
│ zeebe     at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]                                                                                                            │
│ zeebe     at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]                                                                                                                                │
│ zeebe     at java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?]                                                                                                                                │
│ zeebe     at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:750) [spring-boot-3.1.5.jar:3.1.5]                                                                                 │
│ zeebe     at org.springframework.boot.SpringApplication.run(SpringApplication.java:323) [spring-boot-3.1.5.jar:3.1.5]                                                                                         │
│ zeebe     at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:78) [camunda-zeebe-8.3.3.jar:8.3.3]                                                                                          │
│ zeebe Caused by: java.lang.IllegalStateException: Failed to load exporter with configuration: ExporterCfg{, jarPath='/exporters/zeebe-kafka-exporter.jar', className='au.th.zeebe.boundary.CamundaExporter',
zeebe     at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:149) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                          │
│ zeebe     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:70) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                            │
│ zeebe     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:49) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                            │
│ zeebe     at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                                          │
│ zeebe     at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.5.jar:3.1.5]                                                                                 │
│ zeebe     ... 12 more                                                                                                                                                                                         │
│ zeebe Caused by: io.camunda.zeebe.util.jar.ExternalJarLoadException: Cannot load JAR at [/exporters/zeebe-kafka-exporter.jar]: is not readable                                                                │
│ zeebe     at io.camunda.zeebe.util.jar.ExternalJarRepository.verifyJarPath(ExternalJarRepository.java:78) ~[zeebe-util-8.3.3.jar:8.3.3]                                                                       │
│ zeebe     at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:54) ~[zeebe-util-8.3.3.jar:8.3.3]                                                                                │
│ zeebe     at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:47) ~[zeebe-util-8.3.3.jar:8.3.3]                                                                                │
│ zeebe     at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.load(ExporterRepository.java:75) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                        │
│ zeebe     at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:147) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                          │
│ zeebe     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:70) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                            │
│ zeebe     at io.camunda.zeebe.broker.Broker.<init>(Broker.java:49) ~[zeebe-broker-8.3.3.jar:8.3.3]                                                                                                            │
│ zeebe     at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                                          │
│ zeebe     at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.5.jar:3.1.5]                                                                                 │
│ zeebe     ... 12 more                                                                                                                                                                                         │
│ zeebe 2024-01-12 23:28:58.290 [] [main] [] WARN                                                                                                                                                               │
│ zeebe       org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent                         │
│ zeebe java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.broker.Broker.close()" because "this.broker" is null                                                                                    │
│ zeebe     at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:94) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                           │
│ zeebe     at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:34) ~[camunda-zeebe-8.3.3.jar:8.3.3]                                                                           │
│ zeebe     at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:174) ~[spring-context-6.0.13.jar:6.0.13]                             │
│ zeebe     at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:167) ~[spring-context-6.0.13.jar:6.0.13]                               │
│ zeebe     at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) ~[spring-context-6.0.13.jar:6.0.13]                               │
│ zeebe     at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:445) ~[spring-context-6.0.13.jar:6.0.13]                                             │
│ zeebe     at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) ~[spring-context-6.0.13.jar:6.0.13]                                             │
│ zeebe     at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1067) [spring-context-6.0.13.jar:6.0.13]                                                  │
│ zeebe     at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:149) [spring-boot-3.1.5.jar:3.1.5]                            │
│ zeebe     at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1035) [spring-context-6.0.13.jar:6.0.13]                                                    │
│ zeebe     at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:790) [spring-boot-3.1.5.jar:3.1.5]                                                                            │
│ zeebe     at org.springframework.boot.SpringApplication.run(SpringApplication.java:329) [spring-boot-3.1.5.jar:3.1.5]                                                                                         │
│ zeebe     at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:78) [camunda-zeebe-8.3.3.jar:8.3.3]                                                                                          │
│ zeebe 2024-01-12 23:28:58.293 [] [main] [] INFO                                                                                                                                                               │
│ zeebe       org.springframework.boot.web.embedded.netty.GracefulShutdown - Commencing graceful shutdown. Waiting for active requests to complete                                                              │
│ zeebe 2024-01-12 23:28:58.349 [] [netty-shutdown] [] INFO                                                                                                                                                     │
│ Stream closed EOF for bpm/camunda-platform-dev-zeebe-2 (init-exporters-kafka)                                                                                                                                 │
│ zeebe       org.springframework.boot.web.embedded.netty.GracefulShutdown - Graceful shutdown complete                                                                                                         │
│ zeebe 2024-01-12 23:28:58.360 [] [atomix-cluster-0] [] INFO                                                                                                                                                   │
│ zeebe       io.atomix.cluster.AtomixCluster - Stopped                                                                                                                                                         

Updated zeebe config in values file:

zeebe:
  pvcAccessModes:
  - ReadWriteMany
  pvcStorageClassName: "efs-sc"
  extraVolumes:
  - name: exporters-zeebe
    emptyDir: {}
  env:
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_CLASSNAME
    value: au.th.zeebe.boundary.CamundaExporter
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_JARPATH
    value: /exporters/zeebe-kafka-exporter.jar
  initContainers:
  - name: init-exporters-kafka
    image: amazon/aws-cli:2.15.8
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
    command: ["sh", "-c", "aws s3 cp s3://my-bucket/polaris-zeebe-exporter-1.0.0.jar /exporters/zeebe-kafka-exporter.jar && ls -al /exporters"]
    volumeMounts:
    - name: exporters-zeebe
      mountPath: /exporters/
    env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: access-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: secret-access-key   

  extraVolumeMounts:
  - name: exporters-zeebe
    mountPath: /usr/local/zeebe/exporters

[aabouzaid]

The /exporters dir is actually different from the /usr/local/zeebe/exporters.
They shouldn't use the same vol.

Give this a try:

zeebe:
  pvcAccessModes:
  - ReadWriteMany
  pvcStorageClassName: "efs-sc"
  env:
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_CLASSNAME
    value: au.th.zeebe.boundary.CamundaExporter
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_JARPATH
    value: /usr/local/zeebe/exporters/zeebe-kafka-exporter.jar
  initContainers:
  - name: init-exporters-kafka
    image: amazon/aws-cli:2.15.8
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
    command: ["sh", "-c", "aws s3 cp s3://my-bucket/polaris-zeebe-exporter-1.0.0.jar /exporters/zeebe-kafka-exporter.jar && ls -al /exporters"]
    volumeMounts:
    - name: exporters
      mountPath: /exporters
    env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: access-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: secret-access-key   
  extraVolumes:
  - name: exporters-zeebe
    emptyDir: {}
  extraVolumeMounts:
  - name: exporters-zeebe
    mountPath: /usr/local/zeebe/exporters

[aabouzaid]

Closing this issue since no updates in a while.

[aabouzaid]

Also, more details about that topic are on the official docs guide:
https://docs.camunda.io/docs/self-managed/platform-deployment/helm-kubernetes/guides/install-zeebe-exporters/

[npepinpe]

I ran into this today. This fails because the startup.sh script looks into /exporters, and if it finds something, will attempt to mkdir /usr/local/zeebe/exporters, which will fail:

camunda-platform-helm/charts/camunda-platform-latest/templates/zeebe/configmap.yaml

Line 92 in 2f38310

mkdir -p /usr/local/zeebe/exporters/

This looks like a bug to me, but let me know if that's wrong. The workaround AFAIK is to add an extra volume/volume mount to /usr/local/zeebe/exporters and copy your exporters there directly.