-
Notifications
You must be signed in to change notification settings - Fork 308
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Environment Info
Botkube Version: v1.12.0 (also tested v1.14.0)
Kubernetes: EKS v1.32.9
Platform: Socket Slack
Helm Chart: botkube/botkube
Description:
Summary
The reasons filter under event configuration for Kubernetes resources (v1/pods and v1/nodes) does not filter events as expected. All events are sent to Slack regardless of whether the event reason is
included in the reasons list. Additionally, minimumEventCount threshold is also ignored.
Problem
We configured Botkube to only alert for specific event reasons:
- Pods: CrashLoopBackOff, OOMKilled, Failed, ImagePullBackOff, etc.
- Nodes: NodeNotReady, NodeMemoryPressure, NodeDiskPressure, etc.
However, we are still receiving alerts for:
Unhealthy(pod) - NOT in our reasons listInvalidDiskCapacity(node) - NOT in our reasons list
Versions Tested
- Botkube v1.14.0 - Filter NOT working
- Botkube v1.12.0 - Filter NOT working
Environment
- Kubernetes: EKS v1.32.9
- Platform: Socket Slack
- Helm Chart: botkube/botkube
Configuration Used
sources:
k8s-all-events:
botkube/kubernetes:
config:
resources:
- type: v1/pods
event:
types:
- error
- warning
reasons:
- CrashLoopBackOff
- OOMKilled
- Failed
- FailedScheduling
- ImagePullBackOff
- ErrImagePull
minimumEventCount: 5
- type: v1/nodes
event:
types:
- error
- warning
reasons:
- NodeNotReady
- NodeMemoryPressure
- NodeDiskPressure
minimumEventCount: 8
Expected Behavior
1. Only events with reasons in the list should trigger alerts
2. Events with reasons not in the list (like Unhealthy, InvalidDiskCapacity) should be filtered out
3. Events should only alert after reaching minimumEventCount threshold
Actual Behavior
1. ALL events trigger alerts regardless of the reasons filter
2. Unhealthy alerts received even though not in pod reasons list
3. InvalidDiskCapacity alerts received even though not in node reasons list
4. minimumEventCount is ignored - events with count=1 still trigger alerts (threshold was 5)
Steps to Reproduce
1. Deploy Botkube with the configuration above
2. Create a pod with a failing liveness/readiness probe
3. Observe Unhealthy alerts in Slack despite not being in the reasons list
4. Check node events - InvalidDiskCapacity alerts received despite not being in list
Evidence
Pod Unhealthy alert received (not in reasons list):
v1/pods error
Kind: Pod
Reason: Unhealthy
Messages: Liveness probe failed: HTTP probe failed with statuscode: 401
Node InvalidDiskCapacity alert received (not in reasons list):
v1/nodes error
Kind: Node
Reason: InvalidDiskCapacity
Messages: invalid capacity 0 on image filesystem
Impact
Unable to filter noisy alerts, resulting in alert fatigue. The reasons filter appears to have no effect on event filtering.
Evidence of Bug
Expected: Only reasons in the list should trigger alerts
Actual: ALL events trigger alerts regardless of reasons filter
Pod events with reason "Unhealthy" - NOT in list - STILL received alerts
Node events with reason "InvalidDiskCapacity" - NOT in list - STILL received alerts
minimumEventCount: 5 - Events with count=1 still triggered alerts
[botkube-minimal - Copy.yaml](https://github.com/user-attachments/files/24186430/botkube-minimal.-.Copy.yaml)
<!-- Describe what happens instead. -->
## Steps to reproduce
<!-- Steps to reproduce the issue. Attach all resources that can help us understand the bug, such as screenshots, links or files. -->Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working