Add docker-compose setup

Author: stephanpelikan

.env

@@ -18,6 +18,7 @@ An easy to use exporter which will export Zeebe records to a configured Kafka to
 - [Zeebe Kafka Exporter](#zeebe-kafka-exporter)
   * [Supported Zeebe versions](#supported-zeebe-versions)
   * [Backwards compatibility](#backwards-compatibility)
+  * [Docker](#docker)
   * [Quick start](#quick-start)
   * [Usage](#usage)
     + [Kafka configuration](#kafka-configuration)
@@ -33,34 +34,19 @@ An easy to use exporter which will export Zeebe records to a configured Kafka to
   * [Build from source](#build-from-source)
     + [Prerequisites](#prerequisites)
     + [Building](#building)
-  * [Backwards compatibility](#backwards-compatibility-1)
   * [Report issues or contact developers](#report-issues-or-contact-developers)
   * [Create a Pull Request](#create-a-pull-request)
   * [Commit Message Guidelines](#commit-message-guidelines)
   * [Contributor License Agreement](#contributor-license-agreement)
 
 ## Supported Zeebe versions
 
-Version 1.x and 2.x is compatible with the following Zeebe versions:
+Each exporter uses the Zeebe protocol according to Zeebe of the same version.
 
-- 0.23.x
-- 0.24.x
-- 0.25.x
-- 0.26.x
-
-Version 3.x is compatible with the following Zeebe versions:
-
-- 1.0
-
-## Backwards compatibility
-
-As there is currently only a single maintainer, only the latest major version will be maintained and
-supported.
-
-At the moment, the only guarantees for backwards compatibility are:
+### Docker
 
-- the exporter's configuration
-- the serde module
+A ready to use Docker compose setup is provided for local development.
+Checkout [./docker-compose-8.6-kafka](./docker-compose-8.6-kafka/README.md) for details.
 
 ## Quick start
 
@@ -274,58 +260,6 @@ public final class MyClass {
 }
 ```
 
-### Docker
-
-The [docker-compose.yml](/docker-compose.yml) found in the root of the project is a good example of
-how you can deploy Zeebe, Kafka, and connect them via the exporter.
-
-To run it, first build the correct exporter artifact which `docker-compose` can find. From the root
-of the project, run:
-
-```shell
-mvn install -DskipTests -Dexporter.finalName=zeebe-kafka-exporter
-```
-
-> It's important here to note that we set the artifact's final name - this allows us to use a fixed
-> name in the `docker-compose.yml` in order to mount the file to the Zeebe container.
-
-Then you start the services - they can be started in parallel with no worries.
-
-```shell
-docker-compose up -d
-```
-
-> If you wish to stop these containers, remember that some of them create volumes, so unless you
-> plan on reusing those make sure to bring everything down using `docker-compose down -v`.
-
-The services started are the following:
-
-- zeebe: with the gateway port (26500) opened
-- kafka: with the standard 9092 port opened for internal communication, and port 29092 for external
-- consumer: a simple kafkacat image which will print out every record published on any topic
-  starting with `zeebe`
-- zookeeper: required to start Kafka
-
-Once everything is up and running, use your Zeebe cluster as you normally would. For example, given
-a workflow at `~/workflow.bpmn`, you could deploy it as:
-
-```shell
-zbctl --insecure deploy ~/workflow.bpmn
-```
-
-After this, you can see the messages being consumed by the consumer running:
-
-```shell
-docker logs -f consumer
-```
-
-> You may see some initial error logs from the consumer - this happens while the Kafka broker isn't
-> fully up, but it should stop once kafkacat can connect to it.
-
-> The first time a record of a certain kind (e.g. deployment, job, workflow, etc.) is published, it
-> will create a new topic for it. The consumer is refreshing the list of topics every second, which
-> means that for that first message there may be a bit of delay.
-
 ## Reference
 
 The exporter uses a Kafka producer to push records out to different topics based on the incoming

README.md

@@ -0,0 +1,41 @@
+## Image versions ##
+# renovate: datasource=docker depName=camunda/connectors-bundle
+CAMUNDA_CONNECTORS_VERSION=8.6.8
+# renovate: datasource=docker depName=camunda/zeebe
+CAMUNDA_ZEEBE_VERSION=8.6.9
+# renovate: datasource=docker depName=camunda/identity
+CAMUNDA_IDENTITY_VERSION=8.6.8
+# renovate: datasource=docker depName=camunda/operate
+CAMUNDA_OPERATE_VERSION=8.6.9
+# renovate: datasource=docker depName=camunda/tasklist
+CAMUNDA_TASKLIST_VERSION=8.6.9
+# renovate: datasource=docker depName=camunda/optimize
+CAMUNDA_OPTIMIZE_VERSION=8.6.5
+# renovate: datasource=docker depName=camunda/web-modeler-restapi
+CAMUNDA_WEB_MODELER_VERSION=8.6.7
+# renovate: datasource=docker depName=elasticsearch
+ELASTIC_VERSION=8.15.5
+KEYCLOAK_SERVER_VERSION=24.0.5
+# renovate: datasource=docker depName=axllent/mailpit
+MAILPIT_VERSION=v1.20.7
+POSTGRES_VERSION=14.5-alpine
+HOST=localhost
+KEYCLOAK_HOST=localhost
+
+## Configuration ##
+# By default the zeebe api is public, when setting this to `identity` a valid zeebe client token is required
+ZEEBE_AUTHENTICATION_MODE=identity
+ZEEBE_CLIENT_ID=zeebe
+ZEEBE_CLIENT_SECRET=zecret
+
+# Set to 'true' to enable resource based authorizations for users and groups
+# This can be used to limit access for users or groups to view/update specific
+# processes and decisions in Operate and Tasklist
+RESOURCE_AUTHORIZATIONS_ENABLED=false
+
+# Set to 'true' to enable multi-tenancy across all components
+# This requires use of identity for authentication
+#
+#  ZEEBE_AUTHENTICATION_MODE=identity
+#
+MULTI_TENANCY_ENABLED=true

docker-compose-8.6-kafka/.env

@@ -0,0 +1,5 @@
+es:
+  settings:
+    index:
+      number_of_replicas: 0
+

docker-compose-8.6-kafka/.optimize/environment-config.yaml

@@ -0,0 +1 @@
+CAMUNDA_MODELER_CLUSTERS_0_AUTHENTICATION: oauth

docker-compose-8.6-kafka/.web-modeler/cluster-config-authentication-mode-identity.env

@@ -0,0 +1 @@
+CAMUNDA_MODELER_CLUSTERS_0_AUTHENTICATION: none

docker-compose-8.6-kafka/.web-modeler/cluster-config-authentication-mode-none.env

@@ -0,0 +1,15 @@
+# Camunda 8 Self-Managed - Docker Compose
+
+## Usage
+
+For end user usage, please check the offical documentation of [Camunda 8 Self-Managed Docker Compose](https://docs.camunda.io/docs/8.6/self-managed/setup/deploy/local/docker-compose/).
+
+## Changes
+
+In this copy of Camunda platform 8.6.9 these changes have been applied:
+
+1. All containers restart policy is set to `no`.
+1. Tenants enabled.
+1. Kafka 3.7.2 added, configured to listen on localhost:9092.
+1. Kafka exporter added to Zeebe.<br>
+   For topic configuration see [exporter.yml](./exporters/exporter.yml).

docker-compose-8.6-kafka/README.md

@@ -0,0 +1,2 @@
+# add secrets per line in the format NAME=VALUE
+# WARNING: ensure not to commit changes to this file

docker-compose-8.6-kafka/connector-secrets.txt

@@ -0,0 +1,167 @@
+# While the Docker images themselves are supported for production usage,
+# this docker-compose.yaml is designed to be used by developers to run
+# an environment locally. It is not designed to be used in production.
+# We recommend to use Kubernetes in production with our Helm Charts:
+# https://docs.camunda.io/docs/self-managed/platform-deployment/kubernetes-helm/
+# For local development, we recommend using KIND instead of `docker-compose`:
+# https://docs.camunda.io/docs/self-managed/platform-deployment/helm-kubernetes/guides/local-kubernetes-cluster/
+
+# This is a lightweight configuration with Zeebe, Operate, Tasklist, and Elasticsearch
+# See docker-compose.yml for a configuration that also includes Optimize, Identity, and Keycloak.
+
+services:
+
+  zeebe: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
+    image: camunda/zeebe:${CAMUNDA_ZEEBE_VERSION}
+    container_name: zeebe
+    ports:
+      - "26500:26500"
+      - "9600:9600"
+      - "8088:8080"
+    environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/
+      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME=io.camunda.zeebe.exporter.ElasticsearchExporter
+      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL=http://elasticsearch:9200
+      # default is 1000, see here: https://github.com/camunda/zeebe/blob/main/exporters/elasticsearch-exporter/src/main/java/io/camunda/zeebe/exporter/ElasticsearchExporterConfiguration.java#L259
+      - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_SIZE=1
+      # allow running with low disk space
+      - ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
+      - ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
+      - "JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m"
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD-SHELL", "timeout 10s bash -c ':> /dev/tcp/127.0.0.1/9600' || exit 1" ]
+      interval: 30s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    volumes:
+      - zeebe:/usr/local/zeebe/data
+    networks:
+      - camunda-platform
+    depends_on:
+      - elasticsearch
+
+  operate: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#operate
+    image: camunda/operate:${CAMUNDA_OPERATE_VERSION}
+    container_name: operate
+    ports:
+      - "8081:8080"
+    environment: # https://docs.camunda.io/docs/self-managed/operate-deployment/configuration/
+      - CAMUNDA_OPERATE_ZEEBE_GATEWAYADDRESS=zeebe:26500
+      - CAMUNDA_OPERATE_ELASTICSEARCH_URL=http://elasticsearch:9200
+      - CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_URL=http://elasticsearch:9200
+      - CAMUNDA_OPERATE_CSRFPREVENTIONENABLED=false
+      - management.endpoints.web.exposure.include=health
+      - management.endpoint.health.probes.enabled=true
+    healthcheck:
+      test: [ "CMD-SHELL", "wget -O - -q 'http://localhost:9600/actuator/health/readiness'" ]
+      interval: 30s
+      timeout: 1s
+      retries: 5
+      start_period: 30s
+    networks:
+      - camunda-platform
+    depends_on:
+      - zeebe
+      - elasticsearch
+
+  tasklist: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#tasklist
+    image: camunda/tasklist:${CAMUNDA_TASKLIST_VERSION}
+    container_name: tasklist
+    ports:
+      - "8082:8080"
+    environment: # https://docs.camunda.io/docs/self-managed/tasklist-deployment/configuration/
+      - CAMUNDA_TASKLIST_ZEEBE_GATEWAYADDRESS=zeebe:26500
+      - CAMUNDA_TASKLIST_ZEEBE_RESTADDRESS=http://zeebe:8080
+      - CAMUNDA_TASKLIST_ELASTICSEARCH_URL=http://elasticsearch:9200
+      - CAMUNDA_TASKLIST_ZEEBEELASTICSEARCH_URL=http://elasticsearch:9200
+      - CAMUNDA_TASKLIST_CSRFPREVENTIONENABLED=false
+      - management.endpoints.web.exposure.include=health
+      - management.endpoint.health.probes.enabled=true
+    healthcheck:
+      test: [ "CMD-SHELL", "wget -O - -q 'http://localhost:9600/actuator/health/readiness'" ]
+      interval: 30s
+      timeout: 1s
+      retries: 5
+      start_period: 30s
+    networks:
+      - camunda-platform
+    depends_on:
+      - zeebe
+      - elasticsearch
+
+  connectors: # https://docs.camunda.io/docs/components/integration-framework/connectors/out-of-the-box-connectors/available-connectors-overview/
+    image: camunda/connectors-bundle:${CAMUNDA_CONNECTORS_VERSION}
+    container_name: connectors
+    ports:
+      - "8085:8080"
+    environment:
+      - ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS=zeebe:26500
+      - ZEEBE_CLIENT_SECURITY_PLAINTEXT=true
+      - CAMUNDA_OPERATE_CLIENT_URL=http://operate:8080
+      - CAMUNDA_OPERATE_CLIENT_USERNAME=demo
+      - CAMUNDA_OPERATE_CLIENT_PASSWORD=demo
+      - management.endpoints.web.exposure.include=health
+      - management.endpoint.health.probes.enabled=true
+    healthcheck:
+      test: [ "CMD-SHELL", "curl -f http://localhost:8080/actuator/health/readiness" ]
+      interval: 30s
+      timeout: 1s
+      retries: 5
+      start_period: 30s
+    env_file: connector-secrets.txt
+    networks:
+      - camunda-platform
+    depends_on:
+      - zeebe
+      - operate
+
+  elasticsearch: # https://hub.docker.com/_/elasticsearch
+    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
+    container_name: elasticsearch
+    ports:
+      - "9200:9200"
+      - "9300:9300"
+    environment:
+      - bootstrap.memory_lock=true
+      - discovery.type=single-node
+      - xpack.security.enabled=false
+      # allow running with low disk space
+      - cluster.routing.allocation.disk.threshold_enabled=false
+      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD-SHELL", "curl -f http://localhost:9200/_cat/health | grep -q green" ]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+    volumes:
+      - elastic:/usr/share/elasticsearch/data
+    networks:
+      - camunda-platform
+
+  kibana:
+    image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION}
+    container_name: kibana
+    ports:
+      - 5601:5601
+    volumes:
+      - kibana:/usr/share/kibana/data
+    networks:
+      - camunda-platform
+    depends_on:
+      - elasticsearch
+    profiles:
+      - kibana
+
+volumes:
+  zeebe:
+  elastic:
+  kibana:
+
+networks:
+  camunda-platform: