Add metrics and error logging if the MEL FSM is stuck in same state for a long time #3610
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rauljordan
approved these changes
Sep 10, 2025
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## raul/mel-inbox-reading #3610 +/- ##
==========================================================
+ Coverage 24.31% 24.34% +0.03%
==========================================================
Files 395 395
Lines 59767 59773 +6
==========================================================
+ Hits 14532 14553 +21
+ Misses 43003 42976 -27
- Partials 2232 2244 +12 🚀 New features to boost your workflow:
|
joshuacolvin0
requested changes
Sep 17, 2025
arbnode/mel/runner/mel.go
Outdated
| f.Int(prefix+".delayed-message-backlog-capacity", DefaultMessageExtractionConfig.DelayedMessageBacklogCapacity, "target capacity of the delayed message backlog") | ||
| f.Uint64(prefix+".blocks-to-prefetch", DefaultMessageExtractionConfig.BlocksToPrefetch, "the number of blocks to prefetch relevant logs from") | ||
| f.String(prefix+".read-mode", DefaultMessageExtractionConfig.ReadMode, "mode to only read latest or safe or finalized L1 blocks. Enabling safe or finalized disables feed input and output. Defaults to latest. Takes string input, valid strings- latest, safe, finalized") | ||
| f.Uint64(prefix+".stall-tolerance", DefaultMessageExtractionConfig.StallTolerance, "max number of times the MEL fsm is allowed to be stuck in the same state before logging an error and firing the ") |
There was a problem hiding this comment.
Shorten description, max times the MEL fsm is allowed to be stuck without logging error is plenty.
arbnode/mel/runner/mel.go
Outdated
| ) | ||
|
|
||
| var ( | ||
| stuckFSMIndicatingGauge = metrics.NewRegisteredGauge("arb/mel/stuck", nil) // 1-stuck, 0-not_stuck. TODO: once this is merged into master notify SRE to create an alert |
There was a problem hiding this comment.
This TODO has nothing to do with source code, should be removed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a way to detect if MEL has been stuck in the same state for a long time by introducing a new config option
which represents the max number of times MEL is allowed to be stuck before setting the metric
arb/mel/stuck(gauge) to 1 and an error log is emitted as well. This PR also does cleanup of removingretryIntervalfield of mel struct and replacing that with theRetryIntervalfield fromMessageExtractionConfig.Resolves NIT-3392