Skip to content

[Bug]: fault-remediation module is stuck with error :"Failed to watch change stream" after v0.7.1 upgrade #955

@deesharma24

Description

@deesharma24

Prerequisites

  • I searched existing issues
  • I can reproduce this issue

Bug Description

when fault-remediation comes up, it is not able to watch the mongodb changestream and is throwing the below error

{"time":"2026-02-26T11:55:55.346439016Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}

it needs to be checked how best to remediate it.

Component

Health Monitor

Steps to Reproduce

After upgrade to v0.7.1, this issue is coming. This is seen only once and there is no mechanism to repro this yet.

Environment

  • NVSentinel version: V0.7.1
  • Kubernetes version:v1.29.15
  • Deployment method: helm chart

Logs/Output

{"time":"2026-02-26T11:55:55.346433518Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346435161Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346437384Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346439016Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346440729Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346442412Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346444104Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346445787Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}
{"time":"2026-02-26T11:55:55.346447439Z","level":"ERROR","msg":"Failed to watch change stream","module":"fault-remediation","version":"v0.7.1","client":"watcher-factory","error":"(ChangeStreamFatalError) Executor error during getMore :: caused by :: cannot resume stream; the resume token was not found. {_data: \"8269A0042C000000732B042C0100296E5A1004A6C9E0BC444A494DA06DA649B45E45E3463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B65790046645F6964006469A0042CE2A00114E4C9D718000004\"}"}

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions