Skip to content

Commit 39fd905

Browse files
weecoweeco
committed
Fix requirements section
1 parent 7c67aab commit 39fd905

File tree

1 file changed

+43
-46
lines changed

1 file changed

+43
-46
lines changed

‎README.md‎

Lines changed: 43 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,16 @@
1-
# Kafka Minion (Alpha - Still in development)
1+
# Kafka Minion
22

3-
![License](https://img.shields.io/github/license/google-cloud-tools/kafka-minion.svg?color=blue)
4-
[![Go Report Card](https://goreportcard.com/badge/github.com/google-cloud-tools/kafka-minion)](https://goreportcard.com/report/github.com/google-cloud-tools/kafka-minion)
5-
![GitHub release](https://img.shields.io/github/release-pre/google-cloud-tools/kafka-minion.svg)
6-
[![Docker Repository on Quay](https://quay.io/repository/google-cloud-tools/kafka-minion/status "Docker Repository on Quay")](https://quay.io/repository/google-cloud-tools/kafka-minion)
3+
![License](https://img.shields.io/github/license/cloudworkz/kafka-minion.svg?color=blue) [![Go Report Card](https://goreportcard.com/badge/github.com/cloudworkz/kafka-minion)](https://goreportcard.com/report/github.com/cloudworkz/kafka-minion) ![GitHub release](https://img.shields.io/github/release-pre/cloudworkz/kafka-minion.svg) [![Docker Repository on Quay](https://quay.io/repository/cloudworkz/kafka-minion/status "Docker Repository on Quay")](https://quay.io/repository/cloudworkz/kafka-minion)
74

8-
Kafka minion is a prometheus exporter for Apache Kafka (v0.10.0+), created to reliably expose consumer group lag information along with other helpful, unique metrics. Easy to setup on Kubernetes environments.
5+
Kafka minion is a prometheus exporter for Apache Kafka (v0.11.0.2+), created to reliably expose consumer group lag information along with other helpful, unique metrics. Easy to setup on Kubernetes environments.
96

107
![Grafana Dashboard for Kafka Consumer Group Lag Monitoring](https://raw.githubusercontent.com/cloudworkz/kafka-minion/master/grafana-sample-1.png)
118

129
![Grafana Dashboard for Kafka Consumer Group Lag Monitoring](https://raw.githubusercontent.com/cloudworkz/kafka-minion/master/grafana-sample-2.png)
1310

1411
## Features
1512

16-
- [x] Supports Kafka 0.10.1.0 - 2.1.x (last updated 25th Mar 2019)
13+
- [x] Supports Kafka 0.11.0.2 - 2.2.x (last updated 16th Jun 2019)
1714
- [x] Fetches consumer group information directly from `__consumer_offsets` topic instead of querying single brokers for consumer group lags to ensure robustness in case of broker failures or leader elections
1815
- [x] Kafka SASL/SSL support
1916
- [x] Provides per consumergroup:topic lag metrics (removes a topic's metrics if a single partition metric in that topic couldn't be fetched)
@@ -31,26 +28,26 @@ Kafka minion is a prometheus exporter for Apache Kafka (v0.10.0+), created to re
3128

3229
### Environment variables
3330

34-
| Variable name | Description | Default |
35-
| ---------------------------------- | ----------------------------------------------------------------------------------------------------- | -------------------- |
36-
| TELEMETRY_HOST | Host to listen on for the prometheus exporter | 0.0.0.0 |
37-
| TELEMETRY_PORT | HTTP Port to listen on for the prometheus exporter | 8080 |
38-
| LOG_LEVEL | Log granularity (debug, info, warn, error, fatal, panic) | info |
39-
| VERSION | Application version (env variable is set in Dockerfile) | (from Dockerfile) |
40-
| EXPORTER_IGNORE_SYSTEM_TOPICS | Don't expose metrics about system topics (any topic names which are "\_\_" or "\_confluent" prefixed) | true |
41-
| EXPORTER_METRICS_PREFIX | A prefix for all exported prometheus metrics | kafka_minion |
42-
| KAFKA_BROKERS | Array of broker addresses, delimited by comma (e. g. "kafka-1:9092, kafka-2:9092") | (No default) |
43-
| KAFKA_CONSUMER_OFFSETS_TOPIC_NAME | Topic name of topic where kafka commits the consumer offsets | \_\_consumer_offsets |
44-
| KAFKA_SASL_ENABLED | Bool to enable/disable SASL authentication (only SASL_PLAINTEXT is supported) | false |
45-
| KAFKA_SASL_USE_HANDSHAKE | Whether or not to send the Kafka SASL handshake first | true |
46-
| KAFKA_SASL_USERNAME | SASL Username | (No default) |
47-
| KAFKA_SASL_PASSWORD | SASL Password | (No default) |
48-
| KAFKA_TLS_ENABLED | Whether or not to use TLS when connecting to the broker | false |
49-
| KAFKA_TLS_CA_FILE_PATH | Path to the TLS CA file | (No default) |
50-
| KAFKA_TLS_KEY_FILE_PATH | Path to the TLS key file | (No default) |
51-
| KAFKA_TLS_CERT_FILE_PATH | Path to the TLS cert file | (No default) |
52-
| KAFKA_TLS_INSECURE_SKIP_TLS_VERIFY | If true, TLS accepts any certificate presented by the server and any host name in that certificate. | true |
53-
| KAFKA_TLS_PASSPHRASE | Passphrase to decrypt the TLS Key | (No default) |
31+
| Variable name | Description | Default |
32+
| --- | --- | --- |
33+
| TELEMETRY_HOST | Host to listen on for the prometheus exporter | 0.0.0.0 |
34+
| TELEMETRY_PORT | HTTP Port to listen on for the prometheus exporter | 8080 |
35+
| LOG_LEVEL | Log granularity (debug, info, warn, error, fatal, panic) | info |
36+
| VERSION | Application version (env variable is set in Dockerfile) | (from Dockerfile) |
37+
| EXPORTER_IGNORE_SYSTEM_TOPICS | Don't expose metrics about system topics (any topic names which are "\_\_" or "\_confluent" prefixed) | true |
38+
| EXPORTER_METRICS_PREFIX | A prefix for all exported prometheus metrics | kafka_minion |
39+
| KAFKA_BROKERS | Array of broker addresses, delimited by comma (e. g. "kafka-1:9092, kafka-2:9092") | (No default) |
40+
| KAFKA_CONSUMER_OFFSETS_TOPIC_NAME | Topic name of topic where kafka commits the consumer offsets | \_\_consumer_offsets |
41+
| KAFKA_SASL_ENABLED | Bool to enable/disable SASL authentication (only SASL_PLAINTEXT is supported) | false |
42+
| KAFKA_SASL_USE_HANDSHAKE | Whether or not to send the Kafka SASL handshake first | true |
43+
| KAFKA_SASL_USERNAME | SASL Username | (No default) |
44+
| KAFKA_SASL_PASSWORD | SASL Password | (No default) |
45+
| KAFKA_TLS_ENABLED | Whether or not to use TLS when connecting to the broker | false |
46+
| KAFKA_TLS_CA_FILE_PATH | Path to the TLS CA file | (No default) |
47+
| KAFKA_TLS_KEY_FILE_PATH | Path to the TLS key file | (No default) |
48+
| KAFKA_TLS_CERT_FILE_PATH | Path to the TLS cert file | (No default) |
49+
| KAFKA_TLS_INSECURE_SKIP_TLS_VERIFY | If true, TLS accepts any certificate presented by the server and any host name in that certificate. | true |
50+
| KAFKA_TLS_PASSPHRASE | Passphrase to decrypt the TLS Key | (No default) |
5451

5552
### Grafana Dashboard
5653

@@ -78,33 +75,33 @@ Below metrics have a variety of different labels, explained in this section:
7875

7976
#### Consumer group metrics
8077

81-
| Metric | Description |
82-
| --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
83-
| `kafka_minion_group_topic_lag{group, group_base_name, group_is_latest, group_version, topic}` | Number of messages the consumer group is behind for a given topic. |
84-
| `kafka_minion_group_topic_partition_lag{group, group_base_name, group_is_latest, group_version, topic, partition}` | Number of messages the consumer group is behind for a given partition. |
85-
| `kafka_minion_group_topic_partition_offset{group, group_base_name, group_is_latest, group_version, topic, partition}` | Current offset of a given group on a given partition. |
78+
| Metric | Description |
79+
| --- | --- |
80+
| `kafka_minion_group_topic_lag{group, group_base_name, group_is_latest, group_version, topic}` | Number of messages the consumer group is behind for a given topic. |
81+
| `kafka_minion_group_topic_partition_lag{group, group_base_name, group_is_latest, group_version, topic, partition}` | Number of messages the consumer group is behind for a given partition. |
82+
| `kafka_minion_group_topic_partition_offset{group, group_base_name, group_is_latest, group_version, topic, partition}` | Current offset of a given group on a given partition. |
8683
| `kafka_minion_group_topic_partition_commit_count{group, group_base_name, group_is_latest, group_version, topic, partition}` | Number of commited offset entries by a consumer group for a given partition. Helpful to determine the commit rate to possibly tune the consumer performance. |
87-
| `kafka_minion_group_topic_partition_last_commit{group, group_base_name, group_is_latest, group_version, topic, partition}` | Timestamp of last consumer group commit on a given partition |
84+
| `kafka_minion_group_topic_partition_last_commit{group, group_base_name, group_is_latest, group_version, topic, partition}` | Timestamp of last consumer group commit on a given partition |
8885

8986
#### Topic / Partition metrics
9087

91-
| Metric | Description |
92-
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
93-
| `kafka_minion_topic_partition_count{topic, cleanup_policy}` | Partition count for a given topic along with cleanup policy as label |
94-
| `kafka_minion_topic_partition_high_water_mark{topic, partition}` | Latest known commited offset for this partition. This metric is being updated periodically and thus the actual high water mark may be ahead of this one. |
95-
| `kafka_minion_topic_partition_low_water_mark{topic, partition}` | Oldest known commited offset for this partition. This metric is being updated periodically and thus the actual high water mark may be ahead of this one. |
96-
| `kafka_minion_topic_partition_message_count{topic, partition}` | Number of messages for a given partition. Calculated by subtracting high water mark by low water mark. Thus this metric is likely to be invalid for compacting topics, but it still can be helpful to get an idea about the number of messages in that topic. |
88+
| Metric | Description |
89+
| --- | --- |
90+
| `kafka_minion_topic_partition_count{topic, cleanup_policy}` | Partition count for a given topic along with cleanup policy as label |
91+
| `kafka_minion_topic_partition_high_water_mark{topic, partition}` | Latest known commited offset for this partition. This metric is being updated periodically and thus the actual high water mark may be ahead of this one. |
92+
| `kafka_minion_topic_partition_low_water_mark{topic, partition}` | Oldest known commited offset for this partition. This metric is being updated periodically and thus the actual high water mark may be ahead of this one. |
93+
| `kafka_minion_topic_partition_message_count{topic, partition}` | Number of messages for a given partition. Calculated by subtracting high water mark by low water mark. Thus this metric is likely to be invalid for compacting topics, but it still can be helpful to get an idea about the number of messages in that topic. |
9794

9895
#### Internal metrics
9996

100-
| Metric | Description |
101-
| ------------------------------------------------------------------------------- | ----------------------------------------------------------- |
102-
| `kafka_minion_internal_offset_consumer_offset_commits_read{version}` | Number of read offset commit messages |
103-
| `kafka_minion_internal_offset_consumer_offset_commits_tombstones_read{version}` | Number of tombstone messages of all offset commit messages |
104-
| `kafka_minion_internal_offset_consumer_group_metadata_read{version}` | Number of read group metadata messages |
97+
| Metric | Description |
98+
| --- | --- |
99+
| `kafka_minion_internal_offset_consumer_offset_commits_read{version}` | Number of read offset commit messages |
100+
| `kafka_minion_internal_offset_consumer_offset_commits_tombstones_read{version}` | Number of tombstone messages of all offset commit messages |
101+
| `kafka_minion_internal_offset_consumer_group_metadata_read{version}` | Number of read group metadata messages |
105102
| `kafka_minion_internal_offset_consumer_group_metadata_tombstones_read{version}` | Number of tombstone messages of all group metadata messages |
106-
| `kafka_minion_internal_kafka_messages_in_success{topic}` | Number of successfully received kafka messages |
107-
| `kafka_minion_internal_kafka_messages_in_failed{topic}` | Number of errors while consuming kafka messages |
103+
| `kafka_minion_internal_kafka_messages_in_success{topic}` | Number of successfully received kafka messages |
104+
| `kafka_minion_internal_kafka_messages_in_failed{topic}` | Number of errors while consuming kafka messages |
108105

109106
## How does it work
110107

0 commit comments

Comments
 (0)