-
Notifications
You must be signed in to change notification settings - Fork 164
Feat[mqbc, mqbs] FSM mode: Enhance rollover to dynamically change partition max file sizes #1060
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
…luster in case of misconfig Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
…e PR Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
… enhance-rollover
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
… enhance-rollover
Signed-off-by: Aleksandr Ivanov <[email protected]>
… enhance-rollover
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR enhances the rollover procedure in FSM mode to dynamically change partition max file sizes based on storage requirements. When rollover policy is not met and a growth limit is set, partition file sizes can be increased by a configurable step (up to the growth limit). A new RESIZE_STORAGE JournalOp record type is introduced to notify replicas of file size changes before rollover.
Key Changes:
- New configuration parameters:
dataFileGrowLimit,journalFileGrowLimit,qListFileGrowLimit,growStepPercent, andminAvailSpacePercent - New
RESIZE_STORAGEJournalOp record type for communicating file size changes to replicas - Dynamic partition file size adjustment during rollover based on storage requirements
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
src/python/blazingmq/schemas/mqbcfg.py |
Adds new configuration fields for file growth limits and rollover policy parameters |
src/python/blazingmq/dev/it/tweaks/generated.py |
Generates tweak classes for new configuration parameters |
src/python/blazingmq/dev/configurator/__init__.py |
Updates configurator with default values for new parameters |
src/integration-tests/test_fsm_partition_sync.py |
Adds comprehensive integration tests for partition size synchronization and dynamic resizing |
src/groups/mqb/mqbs/mqbs_filestoreprotocol.h |
Major refactoring: adds ResizeStorageData and SyncPointData as nested types in JournalOpRecord, adds maxFileSize to FileHeader |
src/groups/mqb/mqbs/mqbs_filestoreprotocol.cpp |
Updates print methods for new record types and data structures |
src/groups/mqb/mqbs/mqbs_filestore.h |
Adds partitionMaxFileSizes tracking, issueResizeStorage method, and adjustPartitionFileSize helper |
src/groups/mqb/mqbs/mqbs_filestore.cpp |
Implements core rollover logic with dynamic file size adjustment |
src/groups/mqb/mqbc/mqbc_storagemanager.cpp |
Handles RESIZE_STORAGE requests/responses and tracks partition max file sizes across cluster nodes |
src/groups/mqb/mqbc/mqbc_recoverymanager.cpp |
Updates recovery logic to read/write partition max file sizes from file headers |
src/groups/mqb/mqbcfg/mqbcfg.xsd |
Adds new partition configuration elements with default values |
src/groups/bmq/bmqp/bmqp_ctrlmsg.xsd |
Adds PartitionMaxFileSizes type and E_RESIZE replica data type |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
… enhance-rollover
This PR is follow up of #1040.
Rollover procedure is enhanced as follows:
JournalOprecord typeRESIZE_STORAGEis added to notify replicas. This record is issues before sync point ofROLLOVERtype;