Skip to content

Commit 37fbf72

Browse files
committed
datasources: Add kafka descriptions
Related to #1.
1 parent ed17255 commit 37fbf72

File tree

1 file changed

+31
-0
lines changed

1 file changed

+31
-0
lines changed

content/datasources/kafka.md

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Collecting Data from Kafka
2+
3+
## Scenario
4+
5+
[Kafka](http://kafka.apache.org) is a highly distributed messaging system.
6+
7+
You run Kafka as a messaging system and now want to send the messages into various other systems.
8+
9+
Fluentd can setup to collect messages from Kafka. Applications include:
10+
11+
1. Sending Kafka messages into HDFS for analysis
12+
2. Sending Kafka messages into Elasticsearch for analysis
13+
14+
## Setup
15+
16+
1. Download the latest [kafka-fluentd-consumer jar](https://github.com/treasure-data/kafka-fluentd-consumer/releases).
17+
18+
2. Set kafka-fluentd-consumer settings correctly. (See [fluentd-consumer.properties](https://github.com/treasure-data/kafka-fluentd-consumer/blob/master/config/fluentd-consumer.properties) for example.)
19+
20+
3. Open your Fluentd configuration file and add the following lines:
21+
22+
```
23+
<source>
24+
type exec
25+
command java -Dlog4j.configuration=file:///path/to/log4j.properties -jar /path/to/kafka-fluentd-consumer-LATEST_VERSION-all.jar /path/to/config/fluentd-consumer.properties
26+
tag dummy
27+
format json
28+
</source>
29+
```
30+
31+
With the above setup, Fluentd consumes Kafka messages which are specified topics in `fluentd-consumer.properties` via `in_exec` plugin.

0 commit comments

Comments
 (0)