DocDB: Reduce spammy logs in cdc service, tx manager, and rocksdb operations #29350
Conversation
|
|
✅ Deploy Preview for infallible-bardeen-164bc9 ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
austenLacy
force-pushed
the
al-reduce-overly-spammy-logs
branch
from
f7097f8 to
940de07
Compare
|
|
||
| auto get_stream_metadata = GetStream(stream_id, RefreshStreamMapOption::kIfInitiatedState); | ||
| if (!get_stream_metadata.ok()) { | ||
| LOG(WARNING) << "Read invalid stream id: " << stream_id << " for tablet " << tablet_id << ": " |
There was a problem hiding this comment.
We should NOT be seeing too many of this one
There was a problem hiding this comment.
In the past hour, we've seen this log 1.7M times. This is a factor of the number of tablets of which we have around 21k currently. Not all of those tablets are being CDC'd but many thousands are.
Do you have recommendations for what we should look into since we are seeing this log as frequently as we do?
| return; | ||
| } | ||
| YB_LOG_EVERY_N_SECS(WARNING, 1) << "No local transaction status tablet found"; | ||
| YB_LOG_EVERY_N_SECS(WARNING, 10) << "No local transaction status tablet found"; |
There was a problem hiding this comment.
Setting to 10s is fine. But if you see this then it means you dont have enough transaction status tablets and should take action to create more soon .
There was a problem hiding this comment.
All of our transaction status tables are between 200-450 tablets. What's the scaling factor we should consider for these tablets?
hari90
changed the title
3 participants
What
Reduces overly spammy logs that are exacerbated in larger clusters. In a cluster with 100s of tservers some of these logs were logged millions of times per hour.