-
Notifications
You must be signed in to change notification settings - Fork 53
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Prerequisites
- I searched existing issues
- I can reproduce this issue
Bug Description
when fault-remediation comes up, it is not able to watch the mongodb changestream and is throwing the below error
{"time":"2026-02-26T11:55:55.346439016Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
it needs to be checked how best to remediate it.
Component
Health Monitor
Steps to Reproduce
After upgrade to v0.7.1, this issue is coming. This is seen only once and there is no mechanism to repro this yet.
Environment
- NVSentinel version: V0.7.1
- Kubernetes version:v1.29.15
- Deployment method: helm chart
Logs/Output
{"time":"2026-02-26T11:55:55.346433518Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346435161Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346437384Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346439016Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346440729Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346442412Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346444104Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346445787Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346447439Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working