Skip to content

bug(meta/storage): panic on Snapshot notification in compactor observer manager #25724

@hzxa21

Description

@hzxa21

Describe the bug

RisingWave may panic when handling notifications in the observer manager path.
From production log:

  • WARN risingwave_common_service::observer_manager: Stream of notification terminated.
  • thread 'rw-main' (1116) panicked at src/storage/compactor/src/compactor_observer/observer_manager.rs:66:17: invalid notification info: Snapshot

This suggests Snapshot notifications can reach a code path that treats them as invalid and triggers a panic.

Expected behavior

The system should handle Snapshot notifications gracefully (or ignore them safely) without panicking, and the notification stream should remain healthy.

Actual behavior

The notification stream terminates and rw-main panics with invalid notification info: Snapshot.

Impact

  • Potential service disruption due to panic in main runtime thread.
  • Notification processing may stop unexpectedly.

Environment / branch context

  • Repository: risingwavelabs/risingwave
  • Branch where observed: pr-25706

Logs

WARN risingwave_common_service::observer_manager: Stream of notification terminated.
thread 'rw-main' (1116) panicked at src/storage/compactor/src/compactor_observer/observer_manager.rs:66:17:
invalid notification info: Snapshot

References

Notes

Potential area to investigate:

  • Notification type handling in compactor observer manager.
  • Why Snapshot is reaching a branch that treats it as invalid.
  • Whether observer restart/reconnect flow can deliver Snapshot unexpectedly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions