Skip to content

Latest commit

 

History

History
130 lines (105 loc) · 2.95 KB

publish-logs-to-kafka.md

File metadata and controls

130 lines (105 loc) · 2.95 KB
title content_type related_resources products plugins works_on min_version entities tags tldr tools prereqs cleanup
Publish request and response logs to an Apache Kafka topic with the Kafka Log plugin
how_to
text url
Authentication
/gateway/authentication/
gateway
kafka-log
on-prem
konnect
gateway
3.4
plugin
service
route
consumer
authentication
q a
How do I authenticate Consumers with basic authentication?
Create a Consumer with a username and password in the `basicauth_credentials` configuration. Enable the Basic Authentication plugin globally, and authenticate with the base64-encoded Consumer credentials.
deck
entities inline
services routes
example-service
example-route
title content
Apache Kafka
In this tutorial, we'll be using Apache Kafka: 1. Install Apache Kafka. 2. Get the Kafka Docker image: ```sh docker pull apache/kafka-native:4.0.0 docker run -d --name kafka --network kong-quickstart-net -p 9092:9092 apache/kafka-native:4.0.0 ```
inline
title include_content icon_url
Clean up Konnect environment
cleanup/platform/konnect
/assets/icons/gateway.svg
title include_content icon_url
Destroy the {{site.base_gateway}} container
cleanup/products/gateway
/assets/icons/gateway.svg

Create Kafka topic

Save the path to /bin/kafka-topics as an environment variable:

export KAFKA_HOME='</path/to/kafka/>'

Create the kong-log Kafka topic:

$KAFKA_HOME/bin/kafka-topics --create \
    --bootstrap-server localhost:9092 \
    --replication-factor 1 \
    --partitions 10 \
    --topic kong-log

Since the topic can vary in production instances, set the Kafka topic as an environment variable:

export DECK_KAFKA_TOPIC=kong-log

2. Enable the Kafka Log plugin

Set the host.docker.internal as your host. We're using this as our host in this tutorial because {{site.base_gateway}} is using localhost and both {{site.base_gateway}} and Kafka are in Docker containers.

export DECK_HOST=host.docker.internal

Enable the plugin on the example-route we created earlier:

{% entity_examples %} entities: plugins: - name: kafka-log route: example-route config: bootstrap_servers: - host: ${kafka_host} port: 9092 topic: ${kafka_topic} variables: kafka_host: value: $HOST kafka_topic: value: $KAFKA_TOPIC {% endentity_examples %}

Make sample requests

for i in {1..50} ; do curl http://localhost:8000/anything/$i ; done

3. Validate

Verify the contents of the Kafka kong-log topic:

$KAFKA_HOME/bin/kafka-console-consumer \
    --bootstrap-server localhost:9092 \
    --topic kong-log \
    --partition 0 \
    --from-beginning \
    --timeout-ms 1000