Skip to content

Incorrect high_water_mark / low_water_mark values for non-empty topic from KMinion #302

@derkhin

Description

@derkhin

🐞 Issue Title:

Incorrect message count reported for non-empty topic in Redpanda KMinion


🐳 Environment

  • Image: docker.io/redpandadata/kminion:latest
  • Image ID: docker.io/redpandadata/kminion@sha256:6724ee42c6f5fbf4f3bc90ce60f7172eac112c8063d4848d61201f69e7457e1e

📝 Description

For the topic fynd-json-shipment-bag-status, KMinion is reporting:

As a result, the derived message count (high - low) is 0 for all partitions.


🆚 Expected vs. Actual Behavior

  • Expected:
    message_count = high_water_mark - low_water_mark
    This should reflect the actual number of retained messages per partition, consistent with the Kafka UI.

  • Actual:
    Even though cleanup_policy="delete" and new messages are continuously published to the topic, the reported message count remains 0 in all partitions based on Prometheus/Thanos metrics.


📸 Visual Evidence

  • Thanos Query Output - ( how high_water_mark & low_water_mark values are same ? )
Image Image Image ---

✅ Cleanup Policy

  • cleanup_policy="delete"

🔄 Notes

  • The discrepancy persists across multiple scrapes and after publishing fresh messages.
  • Prometheus/Thanos setup and label filters have been verified.

Please let me know if further logs or configuration details are needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions