Claude finished @strawgate's task in 3m 19s —— View job

Recommendation

Create a new OTel Content Pack named kafkametricsreceiver_otel following the established type: content pattern used by apache_otel and docker_otel. This package should provide pre-built Kibana dashboards for visualizing Kafka cluster health metrics (brokers, topics, partitions, consumer groups) collected via the OpenTelemetry kafkametricsreceiver.

Key distinction: This is separate from the in-progress kafka_input_otel package (PR branch claude/issue-17139-20260216-1731), which uses the Kafka receiver to consume telemetry from Kafka topics. The kafkametricsreceiver collects metrics about Kafka infrastructure itself.

Findings

Kafka Metrics Receiver Capabilities

The kafkametricsreceiver collects 16 metrics (11 default + 5 optional) across three scrapers:

Scraper	Default Metrics
`brokers`	`kafka.brokers` (cluster broker count)
`topics`	`kafka.topic.partitions`, `kafka.partition.current_offset`, `kafka.partition.oldest_offset`, `kafka.partition.replicas`, `kafka.partition.replicas_in_sync`
`consumers`	`kafka.consumer_group.lag`, `kafka.consumer_group.lag_sum`, `kafka.consumer_group.members`, `kafka.consumer_group.offset`, `kafka.consumer_group.offset_sum`

Optional metrics (must be explicitly enabled):

kafka.broker.log_retention_period
kafka.topic.log_retention_period, kafka.topic.log_retention_size, kafka.topic.min_insync_replicas, kafka.topic.replication_factor

Existing OTel Content Pack Patterns

Content packs in this repository use type: content and have a minimal structure:

packages/<name>_otel/
├── manifest.yml          # type: content, opentelemetry category
├── changelog.yml
├── LICENSE.txt
├── docs/README.md
├── img/
│   └── logo.svg          # Package icon
│   └── screenshot.png    # Dashboard screenshot(s)
└── kibana/
    └── dashboard/
        └── <package>-<uuid>.json

The manifest uses discovery.datasets to enable auto-installation when matching data is detected:

type: content
categories:
  - observability
  - opentelemetry
  - message_queue          # Kafka-specific
discovery:
  datasets:
    - name: kafkametricsreceiver.otel

Detailed Action Plan

1. Create Package Directory Structure

mkdir -p packages/kafkametricsreceiver_otel/{docs,img,kibana/dashboard}

2. Create `manifest.yml`

format_version: 3.5.0
name: kafkametricsreceiver_otel
title: "Kafka Metrics OpenTelemetry Assets"
version: 0.1.0
source:
  license: "Elastic-2.0"
description: "Visualize Kafka cluster health metrics (brokers, topics, partitions, consumer groups) collected via OpenTelemetry Collector"
type: content
categories:
  - observability
  - message_queue
  - opentelemetry
conditions:
  kibana:
    version: "^9.2.0"
  elastic:
    subscription: "basic"
discovery:
  datasets:
    - name: kafkametricsreceiver.otel
screenshots:
  - src: /img/kafka_otel_overview.png
    title: Kafka Metrics Overview
    size: 600x600
    type: image/png
icons:
  - src: /img/kafka_otel_logo.svg
    title: Kafka OTel logo
    size: 32x32
    type: image/svg+xml
owner:
  github: elastic/obs-infraobs-integrations
  type: elastic

3. Create `changelog.yml`

# newer versions go on top
- version: "0.1.0"
  changes:
    - description: Initial release of Kafka Metrics OpenTelemetry content package
      type: enhancement
      link: https://github.com/elastic/integrations/pull/XXXX

4. Create `docs/README.md`

Document the package following the pattern in packages/apache_otel/docs/README.md:

Overview of what metrics are collected
Requirements (EDOT Collector, Kibana 9.2.0+)
Example collector configuration for kafkametricsreceiver
Link to upstream kafkametricsreceiver documentation

5. Create Kibana Dashboard(s)

Design dashboards visualizing key Kafka health metrics:

Suggested panels:

Panel	Metrics Used	Visualization
Cluster Overview	`kafka.brokers`	Metric
Consumer Group Lag	`kafka.consumer_group.lag`, `kafka.consumer_group.lag_sum`	Time series
Topic Partitions	`kafka.topic.partitions`	Table by topic
Partition Offsets	`kafka.partition.current_offset`, `kafka.partition.oldest_offset`	Time series
Replica Health	`kafka.partition.replicas`, `kafka.partition.replicas_in_sync`	Gauge/comparison
Consumer Group Members	`kafka.consumer_group.members`	Table

The dashboard should filter on data_stream.dataset: kafkametricsreceiver.otel.

6. Create Package Icon

Create img/kafka_otel_logo.svg - a combined Kafka + OTel logo (see existing examples in hostmetrics_input_otel/img/system-otel.svg).

7. Add CODEOWNERS Entry

Add to CODEOWNERS:

/packages/kafkametricsreceiver_otel @elastic/obs-infraobs-integrations

8. Validate and Test

cd packages/kafkametricsreceiver_otel
elastic-package format
elastic-package check
elastic-package build

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
elastic/integrations	#17787 - Add initial yaml dashboard docs	Referenced guide for creating OTel content packs
elastic/integrations	Branch `claude/issue-17139-20260216-1731` with `kafka_input_otel`	Related but different - OTel input for Kafka receiver (consuming telemetry from Kafka topics), not kafkametricsreceiver

Related Files

Repository	File	Relevance	Sections
elastic/integrations	docs/extend/otel-input-packages.md	Guide for OTel input packages (reference for patterns, not directly applicable to content packs)	Full file
elastic/integrations	packages/apache_otel/manifest.yml	Example content pack manifest with `type: content` pattern	Full file
elastic/integrations	packages/docker_otel/manifest.yml	Example content pack manifest with discovery datasets	Full file
elastic/integrations	packages/apache_otel/docs/README.md	Example documentation structure for OTel content packs	Full file

Related Webpages

Name	URL	Relevance
Kafka Metrics Receiver README	https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kafkametricsreceiver	Configuration options, scrapers, and example configs
Kafka Metrics Receiver Documentation	https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kafkametricsreceiver/documentation.md	Complete list of available metrics, types, and units
OpenTelemetry Kafka Monitoring Guide	https://openobserve.ai/blog/how-to-monitor-kafka-with-otel/	General guidance on Kafka monitoring with OTel

Why is Claude responding? | Type @claude to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Add OTel Content Pack for Kafka Metrics Receiver #17818

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions