This project provides a software component which acts as a bridge between MQTT 3.1.1 and an Apache Kafka® cluster. It enables the one-way communication from MQTT to Kafka, allowing MQTT clients to send data to an Apache Kafka cluster. MQTT subscriptions to read data from the Apache Kafka brokers is out of scope.
Download the ZIP or TAR.GZ file from the GitHub release page and unpack it.
Afterwards, edit the config/application.properties file which contains the configuration and the config/topic-mapping-rules.json with the topics' mapping rules.
Once your configuration is ready, start the bridge using:
bin/mqtt_bridge_run.sh --config-file config/application.properties --mapping-rules config/topic-mapping-rules.jsonDownload the ZIP or TAR.GZ file from the GitHub release page and unpack it.
The MQTT Bridge is deployed using a Kubernetes Deployment, and it is configured using a ConfigMap.
The ConfigMap contains the configuration and the topics' mapping rules files.
The files under the install directory are used to deploy the MQTT Bridge on Kubernetes.
To deploy the MQTT Bridge use the following command:
kubectl apply -f ./installTo enable a seamless integration between MQTT and Kafka, the MQTT Bridge provides a way to map MQTT topics to Kafka topics. This mapping is done using a set of predefined patterns and Kafka topic templates. The MQTT Bridge uses these patterns to map MQTT topics to Kafka topics. As a part of the bridge, a Kafka producer will be responsible for producing messages from the MQTT clients to the Kafka Cluster.
The ToMaR is a set of patterns the user provides defining how the MQTT Bridge maps MQTT topic names to Kafka topic names.
A Mapping Rule is a model that contains an MQTT topic pattern, a Kafka topic template, and optionally a Kafka record key.
All the incoming MQTT message's topic should match an MQTT topic pattern in the ToMaR so that the bridge knows in which Kafka topic to produce this message.
This Kafka topic is defined by a template, Kafka Topic template.
However, if the incoming MQTT message's topic does not match any pattern in the ToMaR, the Bridge uses a default Kafka topic where the incoming message will be mapped to.
This Kafka topic is configurable but if it is not specified, the bridge uses the messages_default topic name.
The optional Kafka record key is used to define the key of the Kafka record that will be produced to the Kafka topic.
It is defined by a template as well, and its default value is null.
A valid ToMaR is a JSON file that contains an array of mapping rules.
Each mapping rule is a JSON object that contains two mandatory properties, mqttTopic, kafkaTopic, and one optional kafkaKey.
The following is an example of a valid ToMaR:
[
{
"mqttTopic": "building/(\\w+)/room/(\\d{1,4})/.*",
"kafkaTopic": "building_$1",
"kafkaKey": "room_$2"
},
{
"mqttTopic": "sensors/([^/]+)/data",
"kafkaTopic": "sensor_data"
},
{
"mqttTopic": "sensors.*",
"kafkaTopic": "sensor_others"
},
{
"mqttTopic": "devices/([^/]+)/data/(\b(all|new|old)\b)",
"kafkaTopic": "device_$1_data",
"kafkaKey": "device_$2"
},
{
"mqttTopic": "locations/([^/]+)(?:\\/.*)?$",
"kafkaTopic": "locations",
"kafkaKey": "location_$1"
}
]- The expressions
.*and(?:\\/.*)?$are used to represent the wildcard#, which in turn represents one or more levels in the MQTT topic hierarchy. However, there are some cases that can lead to unexpected behavior when using the these wildcards interchangeably . Therefore, you should note the following:- You cannot use the
.*wildcard in capturing groups. For example, the patternbuilding/(\\w+)/room/(\\d{1,4})/(.*)is invalid because it will capture the whole subtopic of the pattern, and$3placeholder might include/characters, which will lead to an invalid Kafka topic name. - You can use the
.*wildcard without a preceding/character, but it is not equivalent to when it is preceded by/. For example, the patternbuilding.*will matchbuilding,building/room,buildingroom, and so on. On the other hand, the patternbuilding/.*will matchbuilding/room,building/floor/room, and so on, but notbuildingroomandbuilding/. - You can also use the
(?:\\/.*)?$wildcard to match the whole subtopic level of the pattern. For example, the patternlocations/([^/]+)(?:\\/.*)?$will matchlocations/city/luanda/angola,locations/city,locations/city/, and so on. It's literally equivalent tolocations/+/#.
Please note that the(?:\\/.*)?$wildcard is a non-capturing group, which means that it will not be used to replace the placeholders in thekafkaTopicandkafkaKeytemplates. - The
(?:\\/.*)?$wildcard is different from the.*wildcard. For example, the patternsensors.*will match everything aftersensors, seperated with a slash or not. For example,sensors,sensors/,sensorsdata, and so on. On the other hand, the patternsensors(?:\\/.*)?$will matchsensors,sensors/,sensors/data, and so on, but notsensorsdata.
- You cannot use the
- The expression
([^/]+)is used to represent the wildcard+, which in turn represents a single level in the MQTT topic hierarchy. It worth's mentioning that it is the user's responsibility to adhere to the MQTT 3.1.1 naming conventions when defining the MQTT topic patterns.
Placeholders in the kafkaTopic and kafkaKey templates are defined using the $ character followed by a number.
The number represents the index of the capturing group in the mqttTopic pattern. Please note that the index starts from 1 and not 0.
The MQTT Bridge uses the capturing groups in the mqttTopic pattern to positionally extract the values that will be used to replace the placeholders in the kafkaTopic and kafkaKey templates.
Let's go through each rule in the above example to understand how the MQTT Bridge uses these rules to map MQTT topics to Kafka topics:
-
MQTT Topic:
building/(\\w+)/room/(\\d{1,4}).*-> Kafka Topic:building_$1with Kafka Key:room_$2This rule maps MQTT topics of the form
building/{some word}/room/{some number with 4 digits}/#to the Kafka topicbuilding_$1. For example, if the MQTT topic isbuilding/A/room/1003/floor/2it will be mapped to the Kafka topicbuilding_Awith the keyroom_1003. -
MQTT Topic:
sensors/([^/]+)/data-> Kafka Topic:sensor_datawith Kafka Key:nullThis rule maps MQTT topics of the form
sensors/+/datato the Kafka topicsensor_data. For example, if the MQTT topic issensors/temperature/data, it will be mapped to the Kafka topicsensor_data. Because thekafkaKeyis not defined, the key of the Kafka record will benull. -
MQTT Topic:
sensors.*-> Kafka Topic:sensor_otherswith Kafka Key:nullThis rule maps any MQTT topic starting with
sensors/followed by any number of levels in the hierarchy to the Kafka topicsensor_others. For example, if the MQTT topic issensors/temperature/living-room, it will be mapped to the Kafka topicsensor_otherswith the keysensor. Because thekafkaKeyis not defined, the key of the Kafka record will benull. -
MQTT Topic:
devices/([^/]+)/data/(\b(all|new|old)\b)-> Kafka Topic:device_$1_datawith Kafka Key:device_$2This rule maps MQTT topics of the form
devices/{some word}/data/{either all, new or old}to the Kafka topicdevice_$1_data. For example, if the MQTT topic isdevices/thermostat/data/all, it will be mapped to the Kafka topicdevice_thermostat_datawith the keydevice_all. This example also shows the advantage of using regex in themqttTopicpattern. The last capturing group in themqttTopicshould be a word boundary\bfollowed by eitherall,neworoldand anything else will not match the pattern.
The order in which the rules are defined is important.
The MQTT Bridge will use the first rule that matches the MQTT topic.
For example, if the MQTT topic is sensors/temperature/data, it will be mapped to the Kafka topic sensor_data because sensors/([^/]+)/data matches the MQTT topic before sensors/#.
If we swap the positions of the rules, the MQTT Bridge would use the sensors.* to map the MQTT topic to the Kafka topic sensor_others.
The user can configure the MQTT Bridge using an application.properties file.
This section describes the configuration properties that can be used to configure the MQTT Bridge.
The MQTT bridge can be configured using the appropriate prefix.
Following the prefixes for the specific configurations:
bridge.is the prefix used for general configuration of the Bridge.mqtt.is the prefix used for MQTT configuration of the Bridge.kafka.is the prefix used for Kafka configurations of the Bridge.
A valid configuration file should look like this:
# Bridge configuration
bridge.id=my-bridge
bridge.topic.default=default_topic
# MQTT configuration
mqtt.host=0.0.0.0
mqtt.port=1883
mqtt.max.bytes.message=8092
# MQTT SSL/TLS (optional)
#mqtt.ssl.enable=true
#mqtt.ssl.certificate.location=path/to/server-cert.pem
#mqtt.ssl.key.location=path/to/server-key.pem
# Kafka configuration
kafka.bootstrap.servers=localhost:9092Uncomment the mqtt.ssl.* properties to enable TLS/SSL encryption for MQTT connections.
Enabling TLS/SSL does not automatically set the mqtt.port to 8883, you need to set it manually.
The following table describes the configuration properties defined above.
| Setting | Description | Default |
|---|---|---|
| bridge.id | ID of the bridge | null/undefined |
| bridge.topic.default | Topic to be used if no matches with any mapping rules | messages_default |
| mqtt.host | Host address of the MQTT server | 0.0.0.0 |
| mqtt.port | Port number of the MQTT server | 1883 |
| mqtt.max.bytes.message | Max bytes in message for MQTT decoder | 8092 |
| mqtt.ssl.enable | Enable MQTT over TLS | false |
| mqtt.ssl.certificate.location | Path to the MQTT server certificate in PEM format | null/undefined |
| mqtt.ssl.key.location | Path to the MQTT server private key in PEM format | null/undefined |
| mqtt.ssl.certificate | Inline MQTT server certificate in PEM format | null/undefined |
| mqtt.ssl.key | Inline MQTT server private key in PEM format | null/undefined |
| mqtt.ssl.enabled.protocols | Enabled TLS protocol versions | TLSv1.2,TLSv1.3 |
| mqtt.ssl.enabled.cipher.suites | Enabled TLS cipher suites | JDK SSL/TLS engine defaults |
| mqtt.ssl.named.groups | Enabled TLS named groups | JDK SSL/TLS engine defaults |
| kafka.bootstrap.servers | Bootstrap servers for Apache Kafka | localhost:9092 |
| kafka.producer.* | Any Kafka producer configuration (i.e. acks, linger.ms, ...) | Kafka producer defaults |
To accept MQTT connections over TLS, set mqtt.ssl.enable=true and provide a server certificate and private key in PEM format.
PKCS12 and other keystore formats are not supported.
Each of the certificate and the private key can be supplied either as a file path (mqtt.ssl.certificate.location / mqtt.ssl.key.location) or inline (mqtt.ssl.certificate / mqtt.ssl.key).
Inline values take precedence over file-based values; do not set both for the same item.
There is no separate TLS listener: when TLS is enabled, MQTT clients connect to mqtt.host and mqtt.port using TLS (commonly port 8883).
If mqtt.ssl.enabled.protocols is not set, the bridge uses TLSv1.2,TLSv1.3.
If mqtt.ssl.enabled.cipher.suites is not set, the bridge uses the cipher suites provided by the underlying JDK SSL/TLS engine.
If mqtt.ssl.named.groups is not set, the bridge uses the named groups provided by the underlying JDK SSL/TLS engine.
Other than the above properties, the user can also configure the bridge using environment variables.
You can contribute by:
- Raising any issues you find using Strimzi MQTT Bridge
- Fixing issues by opening Pull Requests
- Improving documentation
- Talking about Strimzi Kafka Bridge
All bugs, tasks or enhancements are tracked as GitHub issues. Issues which might be a good start for new contributors are marked with "good-start" label.
The Building Strimzi MQTT Bridge guide describes how to build Strimzi MQTT Bridge and how to test your changes before submitting a patch or opening a PR.
If you want to get in touch with us first before contributing, you can use:
Learn more on how you can contribute on our Join Us page.
Strimzi Kafka Bridge is licensed under the Apache License, Version 2.0
Strimzi MQTT Bridge release archives and container images are published with GitHub artifact attestations containing SLSA build provenance and, for container images, an SPDX SBOM.
The attestations are signed keylessly with Sigstore using the GitHub Actions OIDC identity, so no public key is needed to verify them.
They are created by the shared Strimzi GitHub Actions workflows, so the --signer-repo strimzi/github-actions option is required when verifying them.
You can verify them with the GitHub CLI.
To verify a release archive downloaded from the GitHub release page:
gh attestation verify mqtt-bridge-<version>.tar.gz --repo strimzi/strimzi-mqtt-bridge --signer-repo strimzi/github-actions
Each GitHub release also contains the attestation bundle release-<version>.intoto.jsonl.
You can use it to verify the archive without querying the GitHub Attestation API:
gh attestation verify mqtt-bridge-<version>.tar.gz --bundle release-<version>.intoto.jsonl --repo strimzi/strimzi-mqtt-bridge --signer-repo strimzi/github-actions
Container images are attested per architecture, so use the architecture-specific tag (for example -amd64, -arm64, -ppc64le or -s390x) or its digest instead of the multi-architecture tag.
To verify the build provenance of a container image:
gh attestation verify oci://quay.io/strimzi/mqtt-bridge:<version>-amd64 --repo strimzi/strimzi-mqtt-bridge --signer-repo strimzi/github-actions
To verify the SBOM attestation of a container image:
gh attestation verify oci://quay.io/strimzi/mqtt-bridge:<version>-amd64 --repo strimzi/strimzi-mqtt-bridge --signer-repo strimzi/github-actions --predicate-type https://spdx.dev/Document/v2.3