Search before asking
Fluss version
0.9.0 (latest release)
Please describe the bug 馃悶
When a primary-key table partition is dropped, CoordinatorEventProcessor.processDropPartition removes its CompletedSnapshotStore instances before unregistering the corresponding metric group.
If a later cleanup step throws, the drop event terminates with the snapshot stores already removed while the gauges remain registered. Subsequent Prometheus scrapes invoke these residual gauges, which look up their backing stores through bucketCompletedSnapshotStores. Because the stores no longer exist, the gauge callback throws a NullPointerException.
Every subsequent scrape encounters the same residual state, so Coordinator metrics can remain unavailable until the process is restarted.
There is also an in-flight scrape race. A reporter may already hold a gauge reference when metric-group unregistration begins. Unregistration alone does not make that callback safe because the gauge resolves its store through the mutable manager map.
Reproduction
- Create a partitioned primary-key table and its snapshot-store gauges.
- Delete the partition in ZooKeeper, allowing the watcher, event throttler, and Coordinator event queue to invoke
processDropPartition.
- Inject a failure in a cleanup step after snapshot-store removal but before metric-group removal.
- Wait for the event to finish.
- Scrape both the active registry gauges and gauge references captured by an in-flight scrape.
Solution
Unregister the table or partition metric group before removing its snapshot stores.
Additionally, bind each snapshot gauge directly to the CompletedSnapshotStore created for that bucket instead of resolving it through the manager map on every scrape. This removes the residual registered state and keeps callbacks captured by an in-flight scrape valid during concurrent unregistration.
Are you willing to submit a PR?
Search before asking
Fluss version
0.9.0 (latest release)
Please describe the bug 馃悶
When a primary-key table partition is dropped,
CoordinatorEventProcessor.processDropPartitionremoves itsCompletedSnapshotStoreinstances before unregistering the corresponding metric group.If a later cleanup step throws, the drop event terminates with the snapshot stores already removed while the gauges remain registered. Subsequent Prometheus scrapes invoke these residual gauges, which look up their backing stores through
bucketCompletedSnapshotStores. Because the stores no longer exist, the gauge callback throws aNullPointerException.Every subsequent scrape encounters the same residual state, so Coordinator metrics can remain unavailable until the process is restarted.
There is also an in-flight scrape race. A reporter may already hold a gauge reference when metric-group unregistration begins. Unregistration alone does not make that callback safe because the gauge resolves its store through the mutable manager map.
Reproduction
processDropPartition.Solution
Unregister the table or partition metric group before removing its snapshot stores.
Additionally, bind each snapshot gauge directly to the
CompletedSnapshotStorecreated for that bucket instead of resolving it through the manager map on every scrape. This removes the residual registered state and keeps callbacks captured by an in-flight scrape valid during concurrent unregistration.Are you willing to submit a PR?