Feat[Storagetool]: cluster state ledger (CSL) file search support #558

alexander-e1off · 2025-01-08T13:48:18Z

Enhance storage tool to search records in cluster state ledger (CSL) file.

To search records in CSL file, only CSL file path (--csl-file option) must be passed, journal file path (--journal-path/--journal-path` options must not be present.

Scenarios of BMQStorageTool usage for CSL file

Output summary for CSL file

Example:

bmqstoragetool --csl-file=<path> --summary

Output records from the beginning of CSL file

Example:

bmqstoragetool --csl-file=<path> --csl-from-begin

NOTE: by default search is done from the latest snapshot.

Search and otput only desired record types in CSL file

Example:

bmqstoragetool --csl-file=<path> --csl-record-type=snapshot --csl-record-type=update

NOTE: snapshot, update, commit and ack are supported. Without this option all record types are selected.

Search and otput records details in CSL file

Example:

bmqstoragetool --csl-file=<path> --details

The same search filters could be applied to above scenarios as for journal file search (except --guid filter).

Example of short result output

[ recordType = SNAPSHOT electorTerm = 401 sequenceNumber = 25 timestamp = 1706022776 ][ logId = 37146FB620 offset = 23492436 ]
[ recordType = UPDATE electorTerm = 401 sequenceNumber = 27 timestamp = 1706029885 ][ logId = 37146FB620 offset = 23503460 ]
1 snapshot record(s) found.
1 update record(s) found.
No commit record(s) found.
No ack record(s) found.

Example of detail result output

===================================

    RecordType           : SNAPSHOT
    Offset               : 388
    LogId                : 87EDF15DC0
    ElectorTerm          : 1
    SequenceNumber       : 1
    HeaderWords          : 8
    LeaderAdvisoryWords  : 30
    Timestamp            : 29OCT2024_14:02:54.000000
    Epoch                : 1730210574
    Record               :     [
      choice = [
        leaderAdvisory = [
          sequenceNumber = [
            electorTerm = 1
            sequenceNumber = 1
          ]
          partitions = [
            [
              partitionId = 0
              primaryNodeId = 0
              primaryLeaseId = 2
            ]
          ]
          queues = [
            [
              uri = "bmq://bmq.test.persistent.priority/my-first-queue"
 key = [ 26DACDC974 ]              partitionId = 0
              appIds = [
              ]
            ]
          ]
        ]
      ]
    ]

=================================

    RecordType           : COMMIT
    Offset               : 540
    LogId                : 87EDF15DC0
    ElectorTerm          : 1
    SequenceNumber       : 2
    HeaderWords          : 8
    LeaderAdvisoryWords  : 10
    Timestamp            : 29OCT2024_14:02:54.000000
    Epoch                : 1730210574
    Record               :     [
      choice = [
        leaderAdvisoryCommit = [
          sequenceNumber = [
            electorTerm = 1
            sequenceNumber = 2
          ]
          sequenceNumberCommitted = [
            electorTerm = 1
            sequenceNumber = 1
          ]
        ]
      ]
    ]

1 snapshot record(s) found.
No update record(s) found.
1 commit record(s) found.
No ack record(s) found.

Example of summary result output

1 snapshot record(s) found.

2 update record(s) found, including:
    queueAssignmentAdvisory  : 1
    queueUnassignedAdvisory  : 1

3 commit record(s) found.

No ack record(s) found.

2 Queues found:
[ uri = "bmq://bmq.test.persistent.fanout/my2" key = [ 676EC94E89 ]
 partitionId = 0 appIds = [ [ appId = "foo" appKey = [ 40D805FBE4 ] ] [ appId = "bar" appKey = [ 8377729B23 ] ] [ appId = "baz" appKey = [ 0C996AF949 ] ] ] ]
[ uri = "bmq://bmq.test.persistent.fanout/my1" key = [ 44D469EF30 ]
 partitionId = 0 appIds = [ [ appId = "bar" appKey = [ 9FC222C73E ] ] [ appId = "baz" appKey = [ 935B5C5FC4 ] ] [ appId = "foo" appKey = [ 489D904F25 ] ] ] ]

Signed-off-by: Aleksandr Ivanov <[email protected]>

678098

Reviewed a part of the PR, have comments

678098 · 2025-03-26T09:40:36Z

src/applications/bmqstoragetool/README.md

+files. It allows to search records in:
+- `journal` file (.bmq_journal) with the set of different filters and output found results in 
+    the short or detail form;
+- `cluster state ledger` (CSL) file (*.bmq_csl) with the set of different filters and output 
+    found results in the short or detail form;  


I think the current form of explanation is a bit too heavy and should be simplified and disambiguated:

Suggested change

files. It allows to search records in:

- `journal` file (.bmq_journal) with the set of different filters and output found results in

the short or detail form;

- `cluster state ledger` (CSL) file (*.bmq_csl) with the set of different filters and output

found results in the short or detail form;

files. Using a set of different filters, it allows to search records in:

- `journal` file (.bmq_journal).

- `cluster state ledger` (CSL) file (*.bmq_csl).

The output results can be returned in either short or detailed form.

678098 · 2025-03-26T09:41:36Z

src/applications/bmqstoragetool/README.md

+    the short or detail form;
+- `cluster state ledger` (CSL) file (*.bmq_csl) with the set of different filters and output 
+    found results in the short or detail form;  
+    NOTE: For this mode, `journal` file (.bmq_journal) **must not** be passed.


If this is a must, it is better to enforce it in the storage tool itself. Also this note can be merged with the next paragraph

678098 · 2025-03-26T09:48:49Z

src/applications/bmqstoragetool/README.md

+Filter messages with corresponding composite sequence numbers (defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Can make this header shorter by moving the note

Suggested change

Filter messages with corresponding composite sequence numbers (defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Filter messages with corresponding composite sequence numbers

-------------------------------------------------------------

Composite sequence numbers are defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file.

678098 · 2025-03-26T12:30:42Z

src/applications/bmqstoragetool/m_bmqstoragetool_compositesequencenumber.cpp

+    const bsls::Types::Uint64 leaseId,
+    const bsls::Types::Uint64 sequenceNumber)


Suggested change

const bsls::Types::Uint64 leaseId,

const bsls::Types::Uint64 sequenceNumber)

bsls::Types::Uint64 leaseId,

bsls::Types::Uint64 sequenceNumber)

Basic arithmetic types passed by value

678098 · 2025-03-26T12:30:59Z

src/applications/bmqstoragetool/m_bmqstoragetool_compositesequencenumber.h

+    explicit CompositeSequenceNumber(const bsls::Types::Uint64 leaseId,
+                                     const bsls::Types::Uint64 sequenceNumber);


Suggested change

explicit CompositeSequenceNumber(const bsls::Types::Uint64 leaseId,

const bsls::Types::Uint64 sequenceNumber);

explicit CompositeSequenceNumber(bsls::Types::Uint64 leaseId,

bsls::Types::Uint64 sequenceNumber);

678098 · 2025-03-26T12:52:56Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslrecordprinter.h

+
+  public:
+    // CREATORS
+    CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator);


Suggested change

CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator);

explicit CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator = 0);

nested traits using bslma allocator

678098 · 2025-03-26T12:54:11Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslrecordprinter.h

+    const mqbsi::LedgerRecordId&          recId)
+{
+    d_fields.clear();
+    d_fields.reserve(9);  // max number of fields


Suggested change

d_fields.reserve(9); // max number of fields

d_fields.reserve(10); // max number of fields

There is 9 or 10 fields depending on if branch triggering

678098 · 2025-03-26T12:57:09Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslrecordprinter.cpp

+
+namespace BloombergLP {
+namespace m_bmqstoragetool {
+


Let's define destructors here for all classes defined in the header to avoid warnings during compilation

I checked build logs on CI for both ubuntu and darwin and see no warnings for these classes. CslRecordPrinter is a template class and should be defined in the header. As I know it make sense for virtual classes.

I checked build logs on CI for both ubuntu and darwin and see no warnings for these classes. CslRecordPrinter is a template class and should be defined in the header. As I know it make sense for virtual classes.

Good point

678098 · 2025-03-26T13:00:49Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslprinter.cpp

+
+  public:
+    // CREATORS
+    HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)


Suggested change

HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)

explicit HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator = 0)

nested trait uses bslma allocator

678098 · 2025-03-26T13:02:27Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslprinter.cpp

+  public:
+    // CREATORS
+
+    JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)


Suggested change

JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)

explicit JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator = 0)

nested trait uses bslma allocator

Signed-off-by: Aleksandr Ivanov <[email protected]>

kaikulimu · 2025-05-07T13:39:29Z

I will give it a try and leave comments

kaikulimu · 2025-05-07T21:13:28Z

@alexander-e1off Some feedback:

Summary has bug with filters

Say a CSL file has an initial snapshot record with zero queues, followed by two queue assignment update records. All records have corresponding commit. If the user runs /bmqstoragetool.tsk --csl-file=<path> --csl-record-type=snapshot --summary, it will show the following:

1 snapshot record(s) found.

2 Queues found:
[ uri = "bmq://bmq.test.mmap.fanout/q2" key = [ 6468D6506E ]
 partitionId = 1 appIds = [ ] ]
[ uri = "bmq://bmq.test.mmap.fanout/q1" key = [ 18F848CC18 ]
 partitionId = 0 appIds = [ ] ]

It says 2 queues found in the summary, but those information did not come solely from the snapshot-type records. We observe similar issue if we run /bmqstoragetool.tsk --csl-file=<path> --csl-record-type=ack --summary:

No ack record(s) found.

2 Queues found:
[ uri = "bmq://bmq.test.mmap.fanout/q2" key = [ 6468D6506E ]
 partitionId = 1 appIds = [ ] ]
[ uri = "bmq://bmq.test.mmap.fanout/q1" key = [ 18F848CC18 ]
 partitionId = 0 appIds = [ ] ]

This is confusing to the user. We have two options:

(Easier but sufficient) Prohibit filtering by record type, and maybe some other filters, while we are in summary mode. Summaries should be short enough that they won't be too long even if filters are prohibited.
Enhance summary mode to show summary over only the filtered in records.

kaikulimu

See above comment

Signed-off-by: Aleksandr Ivanov <[email protected]>

alexander-e1off · 2025-05-08T08:27:10Z

@kaikulimu, thanks for your feedback. Agree that summary should cover all content of CSL file. I've fixed it with option 1:

./bmqstoragetool.tsk --csl-file=<path> --summary --csl-record-type=snapshot
Arguments validation failed:
'--summary' can't be combined with '--details' and filters (record type, range, queue name/key, offset, seq.num) options, as it calculates and outputs statistics

kaikulimu

lgtm

kaikulimu · 2025-05-09T20:45:14Z

@678098 Your review status is "Requested changes", so we can't merge this PR yet. Please review again, and approve and merge if everything looks good.

Signed-off-by: Aleksandr Ivanov <[email protected]>

alexander-e1off added 30 commits November 26, 2024 15:42

Add seqnum and offset input args, add composite sequence number support

4438797

Signed-off-by: Aleksandr Ivanov <[email protected]>

Debug binary search for all range types

465076c

Signed-off-by: Aleksandr Ivanov <[email protected]>

Reemove debug

32f36b3

Signed-off-by: Aleksandr Ivanov <[email protected]>

Simplify getValue template

e0cf5a1

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add search result decorators for seq.number and offset, add UTs

bd9001d

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix hasCache() call

cff0b26

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix code style

1fd8eac

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix code style

81ba90f

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add search for specific sequence numbers and offsets

8a5d825

Signed-off-by: Aleksandr Ivanov <[email protected]>

Cleanup

fd3f96b

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix README.md

42e6b9a

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix typo in README.md

2343748

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update header caption

5189f9c

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix review comments

80c58ce

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix code style

9175303

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add support of queueOp and journalOp records

ec47845

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add UTs, update footer printing

8a46f56

Signed-off-by: Aleksandr Ivanov <[email protected]>

Move printRecord methods to RecordPrinter namespace

d270645

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add printRecord method for queueOp record

98655ac

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add flag to stop search when reached higher bound

3e9f3f4

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update README.md

9be7be8

Signed-off-by: Aleksandr Ivanov <[email protected]>

Revert "Add flag to stop search when reached higher bound"

13522a6

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add flag to stop search when reached higher bound

096a3fa

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update README.md, fix code style

77a57a5

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix typo

062979d

Signed-off-by: Aleksandr Ivanov <[email protected]>

Cosmetic change: move validators to CommandLineArguments

05192f0

Signed-off-by: Aleksandr Ivanov <[email protected]>

Cleanup

a82b95f

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update README.md

d69dbf6

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix comment

15cc481

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix typo

042b1ff

Signed-off-by: Aleksandr Ivanov <[email protected]>

alexander-e1off added 3 commits March 14, 2025 10:11

Merge from main, fix merge conflicts

d951bc2

Signed-off-by: Aleksandr Ivanov <[email protected]>

Merge from main, fix conflicts

fe0bced

Signed-off-by: Aleksandr Ivanov <[email protected]>

Merge from main, fix conflicts

c79785b

Signed-off-by: Aleksandr Ivanov <[email protected]>

678098 requested changes Mar 26, 2025

View reviewed changes

alexander-e1off added 7 commits March 27, 2025 12:39

Fix review comments

4153fd5

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix code formatting

c883ff4

Signed-off-by: Aleksandr Ivanov <[email protected]>

Merge remote-tracking branch 'upstream/main' into csl-file-support

0cd1dd4

Merge remote-tracking branch 'upstream/main' into csl-file-support

bccbdf3

Merge remote-tracking branch 'upstream/main' into csl-file-support

6cb9945

Merge branch 'main' into csl-file-support

bcf4415

Merge branch 'main' into csl-file-support

5c2b41a

kaikulimu self-requested a review May 7, 2025 13:39

Merge remote-tracking branch 'upstream/main' into csl-file-support

e4e3959

kaikulimu assigned alexander-e1off and unassigned 678098 May 7, 2025

kaikulimu requested changes May 7, 2025

View reviewed changes

alexander-e1off added 2 commits May 8, 2025 11:12

Disable filters for '--summary' option

0a75871

Signed-off-by: Aleksandr Ivanov <[email protected]>

Merge remote-tracking branch 'upstream/main' into csl-file-support

603c56f

alexander-e1off assigned kaikulimu and unassigned alexander-e1off May 8, 2025

kaikulimu previously approved these changes May 9, 2025

View reviewed changes

alexander-e1off added 2 commits May 12, 2025 11:36

Fix MSAN issue in cslfileprocessor test

66dc90f

Signed-off-by: Aleksandr Ivanov <[email protected]>

Merge remote-tracking branch 'upstream/main' into csl-file-support

cb3d2e9

alexander-e1off dismissed kaikulimu’s stale review via cb3d2e9 May 12, 2025 08:36

678098 approved these changes May 12, 2025

View reviewed changes

kaikulimu merged commit 214d138 into bloomberg:main May 12, 2025
34 checks passed

		Filter messages with corresponding composite sequence numbers (defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file)
		-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

		const bsls::Types::Uint64 leaseId,
		const bsls::Types::Uint64 sequenceNumber)

		explicit CompositeSequenceNumber(const bsls::Types::Uint64 leaseId,
		const bsls::Types::Uint64 sequenceNumber);

	CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator);
	explicit CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator = 0);

	d_fields.reserve(9); // max number of fields
	d_fields.reserve(10); // max number of fields

	HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)
	explicit HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator = 0)

	JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)
	explicit JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator = 0)


		namespace BloombergLP {
		namespace m_bmqstoragetool {

Feat[Storagetool]: cluster state ledger (CSL) file search support #558

Feat[Storagetool]: cluster state ledger (CSL) file search support #558

Uh oh!

Conversation

alexander-e1off commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scenarios of BMQStorageTool usage for CSL file

Output summary for CSL file

Output records from the beginning of CSL file

Search and otput only desired record types in CSL file

Search and otput records details in CSL file

Example of short result output

Example of detail result output

Example of summary result output

Uh oh!

678098 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaikulimu commented May 7, 2025

Uh oh!

kaikulimu commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary has bug with filters

Uh oh!

kaikulimu left a comment

Choose a reason for hiding this comment

Uh oh!

alexander-e1off commented May 8, 2025

Uh oh!

kaikulimu left a comment

Choose a reason for hiding this comment

Uh oh!

kaikulimu commented May 9, 2025

Uh oh!

Uh oh!

Uh oh!

alexander-e1off commented Jan 8, 2025 •

edited

Loading

kaikulimu commented May 7, 2025 •

edited

Loading