Skip to content

Conversation

@alexander-e1off
Copy link
Collaborator

This PR is follow up of #1040.
Rollover procedure is enhanced as follows:

  • If grow limit is not set (set to zero or equal to max file size), behaviour is not changed;
  • If rollover policy is not met and grow limit is set, partition file size is increased by grow step (up to grow limit) to meet rollover policy;
  • New JournalOp record type RESIZE_STORAGE is added to notify replicas. This record is issues before sync point of ROLLOVER type;
  • During next rollover, partition min file size is calculated to meet rollover policy (file size could be decreased).

Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
…luster in case of misconfig

Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Signed-off-by: Aleksandr Ivanov <[email protected]>
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the rollover procedure in FSM mode to dynamically change partition max file sizes based on storage requirements. When rollover policy is not met and a growth limit is set, partition file sizes can be increased by a configurable step (up to the growth limit). A new RESIZE_STORAGE JournalOp record type is introduced to notify replicas of file size changes before rollover.

Key Changes:

  • New configuration parameters: dataFileGrowLimit, journalFileGrowLimit, qListFileGrowLimit, growStepPercent, and minAvailSpacePercent
  • New RESIZE_STORAGE JournalOp record type for communicating file size changes to replicas
  • Dynamic partition file size adjustment during rollover based on storage requirements

Reviewed changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/python/blazingmq/schemas/mqbcfg.py Adds new configuration fields for file growth limits and rollover policy parameters
src/python/blazingmq/dev/it/tweaks/generated.py Generates tweak classes for new configuration parameters
src/python/blazingmq/dev/configurator/__init__.py Updates configurator with default values for new parameters
src/integration-tests/test_fsm_partition_sync.py Adds comprehensive integration tests for partition size synchronization and dynamic resizing
src/groups/mqb/mqbs/mqbs_filestoreprotocol.h Major refactoring: adds ResizeStorageData and SyncPointData as nested types in JournalOpRecord, adds maxFileSize to FileHeader
src/groups/mqb/mqbs/mqbs_filestoreprotocol.cpp Updates print methods for new record types and data structures
src/groups/mqb/mqbs/mqbs_filestore.h Adds partitionMaxFileSizes tracking, issueResizeStorage method, and adjustPartitionFileSize helper
src/groups/mqb/mqbs/mqbs_filestore.cpp Implements core rollover logic with dynamic file size adjustment
src/groups/mqb/mqbc/mqbc_storagemanager.cpp Handles RESIZE_STORAGE requests/responses and tracks partition max file sizes across cluster nodes
src/groups/mqb/mqbc/mqbc_recoverymanager.cpp Updates recovery logic to read/write partition max file sizes from file headers
src/groups/mqb/mqbcfg/mqbcfg.xsd Adds new partition configuration elements with default values
src/groups/bmq/bmqp/bmqp_ctrlmsg.xsd Adds PartitionMaxFileSizes type and E_RESIZE replica data type

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants