Feat[Storagetool]: cluster state ledger (CSL) file search support #558

alexander-e1off · 2025-01-08T13:48:18Z

Enhance storage tool to search records in cluster state ledger (CSL) file.

To search records in CSL file, only CSL file path (--csl-file option) must be passed, journal file path (--journal-path/--journal-path` options must not be present.

Scenarios of BMQStorageTool usage for CSL file

Output summary for CSL file

Example:

bmqstoragetool --csl-file=<path> --summary

Output records from the beginning of CSL file

Example:

bmqstoragetool --csl-file=<path> --csl-from-begin

NOTE: by default search is done from the latest snapshot.

Search and otput only desired record types in CSL file

Example:

bmqstoragetool --csl-file=<path> --csl-record-type=snapshot --csl-record-type=update

NOTE: snapshot, update, commit and ack are supported. Without this option all record types are selected.

Search and otput records details in CSL file

Example:

bmqstoragetool --csl-file=<path> --details

The same search filters could be applied to above scenarios as for journal file search (except --guid filter).

Example of short result output

[ recordType = SNAPSHOT electorTerm = 401 sequenceNumber = 25 timestamp = 1706022776 ][ logId = 37146FB620 offset = 23492436 ]
[ recordType = UPDATE electorTerm = 401 sequenceNumber = 27 timestamp = 1706029885 ][ logId = 37146FB620 offset = 23503460 ]
1 snapshot record(s) found.
1 update record(s) found.
No commit record(s) found.
No ack record(s) found.

Example of detail result output

===================================

    RecordType           : SNAPSHOT
    Offset               : 388
    LogId                : 87EDF15DC0
    ElectorTerm          : 1
    SequenceNumber       : 1
    HeaderWords          : 8
    LeaderAdvisoryWords  : 30
    Timestamp            : 29OCT2024_14:02:54.000000
    Epoch                : 1730210574
    Record               :     [
      choice = [
        leaderAdvisory = [
          sequenceNumber = [
            electorTerm = 1
            sequenceNumber = 1
          ]
          partitions = [
            [
              partitionId = 0
              primaryNodeId = 0
              primaryLeaseId = 2
            ]
          ]
          queues = [
            [
              uri = "bmq://bmq.test.persistent.priority/my-first-queue"
 key = [ 26DACDC974 ]              partitionId = 0
              appIds = [
              ]
            ]
          ]
        ]
      ]
    ]

=================================

    RecordType           : COMMIT
    Offset               : 540
    LogId                : 87EDF15DC0
    ElectorTerm          : 1
    SequenceNumber       : 2
    HeaderWords          : 8
    LeaderAdvisoryWords  : 10
    Timestamp            : 29OCT2024_14:02:54.000000
    Epoch                : 1730210574
    Record               :     [
      choice = [
        leaderAdvisoryCommit = [
          sequenceNumber = [
            electorTerm = 1
            sequenceNumber = 2
          ]
          sequenceNumberCommitted = [
            electorTerm = 1
            sequenceNumber = 1
          ]
        ]
      ]
    ]

1 snapshot record(s) found.
No update record(s) found.
1 commit record(s) found.
No ack record(s) found.

Example of summary result output

1 snapshot record(s) found.

2 update record(s) found, including:
    queueAssignmentAdvisory  : 1
    queueUnassignedAdvisory  : 1

3 commit record(s) found.

No ack record(s) found.

2 Queues found:
[ uri = "bmq://bmq.test.persistent.fanout/my2" key = [ 676EC94E89 ]
 partitionId = 0 appIds = [ [ appId = "foo" appKey = [ 40D805FBE4 ] ] [ appId = "bar" appKey = [ 8377729B23 ] ] [ appId = "baz" appKey = [ 0C996AF949 ] ] ] ]
[ uri = "bmq://bmq.test.persistent.fanout/my1" key = [ 44D469EF30 ]
 partitionId = 0 appIds = [ [ appId = "bar" appKey = [ 9FC222C73E ] ] [ appId = "baz" appKey = [ 935B5C5FC4 ] ] [ appId = "foo" appKey = [ 489D904F25 ] ] ] ]

Signed-off-by: Aleksandr Ivanov <[email protected]>

bbpetukhov · 2025-03-06T15:40:02Z

src/applications/bmqstoragetool/README.md

                        [-h|help]
 Where:
  -r | --record-type          <record type>
-          record type to search {message|queue-op|journal-op} (default: message)
+          record type to search {<message>|queue-op|journal-op} (default: message)


Do we need angle brackets for message?

Just follow the style of bmqtool to highlight default value in case when it is not explicitly defined in balcl::OptionInfo.

src/applications/bmqstoragetool/m_bmqstoragetool_cslfileprocessor.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_cslfileprocessor.h

src/applications/bmqstoragetool/m_bmqstoragetool_cslprinter.cpp

bbpetukhov · 2025-03-06T20:33:37Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslrecordprinter.h

+{
+    d_fields.clear();
+    d_fields.reserve(9);  // max number of fields
+    d_fields.push_back("RecordType");


Consider using emplace_back

Just follow common bmqu::AlignedPrinter usage, used in codebase. d_fields is std::vector<const char*> type, and in this case push_back and emplace_back are equivalent - both just store the pointer.

src/applications/bmqstoragetool/m_bmqstoragetool_cslsearchresult.h

src/applications/bmqstoragetool/m_bmqstoragetool_filters.cpp

Signed-off-by: Aleksandr Ivanov <[email protected]>

678098

Reviewed a part of the PR, have comments

678098 · 2025-03-26T09:40:36Z

src/applications/bmqstoragetool/README.md

+files. It allows to search records in:
+- `journal` file (.bmq_journal) with the set of different filters and output found results in 
+    the short or detail form;
+- `cluster state ledger` (CSL) file (*.bmq_csl) with the set of different filters and output 
+    found results in the short or detail form;  


I think the current form of explanation is a bit too heavy and should be simplified and disambiguated:

Suggested change

files. It allows to search records in:

- `journal` file (.bmq_journal) with the set of different filters and output found results in

the short or detail form;

- `cluster state ledger` (CSL) file (*.bmq_csl) with the set of different filters and output

found results in the short or detail form;

files. Using a set of different filters, it allows to search records in:

- `journal` file (.bmq_journal).

- `cluster state ledger` (CSL) file (*.bmq_csl).

The output results can be returned in either short or detailed form.

678098 · 2025-03-26T09:41:36Z

src/applications/bmqstoragetool/README.md

+    the short or detail form;
+- `cluster state ledger` (CSL) file (*.bmq_csl) with the set of different filters and output 
+    found results in the short or detail form;  
+    NOTE: For this mode, `journal` file (.bmq_journal) **must not** be passed.


If this is a must, it is better to enforce it in the storage tool itself. Also this note can be merged with the next paragraph

678098 · 2025-03-26T09:48:49Z

src/applications/bmqstoragetool/README.md

+Filter messages with corresponding composite sequence numbers (defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Can make this header shorter by moving the note

Suggested change

Filter messages with corresponding composite sequence numbers (defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Filter messages with corresponding composite sequence numbers

-------------------------------------------------------------

Composite sequence numbers are defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file.

678098 · 2025-03-26T12:30:42Z

src/applications/bmqstoragetool/m_bmqstoragetool_compositesequencenumber.cpp

+    const bsls::Types::Uint64 leaseId,
+    const bsls::Types::Uint64 sequenceNumber)


Suggested change

const bsls::Types::Uint64 leaseId,

const bsls::Types::Uint64 sequenceNumber)

bsls::Types::Uint64 leaseId,

bsls::Types::Uint64 sequenceNumber)

Basic arithmetic types passed by value

678098 · 2025-03-26T12:30:59Z

src/applications/bmqstoragetool/m_bmqstoragetool_compositesequencenumber.h

+    explicit CompositeSequenceNumber(const bsls::Types::Uint64 leaseId,
+                                     const bsls::Types::Uint64 sequenceNumber);


Suggested change

explicit CompositeSequenceNumber(const bsls::Types::Uint64 leaseId,

const bsls::Types::Uint64 sequenceNumber);

explicit CompositeSequenceNumber(bsls::Types::Uint64 leaseId,

bsls::Types::Uint64 sequenceNumber);

678098 · 2025-03-26T12:52:56Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslrecordprinter.h

+
+  public:
+    // CREATORS
+    CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator);


Suggested change

CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator);

explicit CslRecordPrinter(bsl::ostream& stream, bslma::Allocator* allocator = 0);

nested traits using bslma allocator

678098 · 2025-03-26T12:54:11Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslrecordprinter.h

+    const mqbsi::LedgerRecordId&          recId)
+{
+    d_fields.clear();
+    d_fields.reserve(9);  // max number of fields


Suggested change

d_fields.reserve(9); // max number of fields

d_fields.reserve(10); // max number of fields

There is 9 or 10 fields depending on if branch triggering

678098 · 2025-03-26T12:57:09Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslrecordprinter.cpp

+
+namespace BloombergLP {
+namespace m_bmqstoragetool {
+


Let's define destructors here for all classes defined in the header to avoid warnings during compilation

I checked build logs on CI for both ubuntu and darwin and see no warnings for these classes. CslRecordPrinter is a template class and should be defined in the header. As I know it make sense for virtual classes.

I checked build logs on CI for both ubuntu and darwin and see no warnings for these classes. CslRecordPrinter is a template class and should be defined in the header. As I know it make sense for virtual classes.

Good point

678098 · 2025-03-26T13:00:49Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslprinter.cpp

+
+  public:
+    // CREATORS
+    HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)


Suggested change

HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)

explicit HumanReadableCslPrinter(bsl::ostream& os, bslma::Allocator* allocator = 0)

nested trait uses bslma allocator

678098 · 2025-03-26T13:02:27Z

src/applications/bmqstoragetool/m_bmqstoragetool_cslprinter.cpp

+  public:
+    // CREATORS
+
+    JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)


Suggested change

JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator)

explicit JsonCslPrinter(bsl::ostream& os, bslma::Allocator* allocator = 0)

nested trait uses bslma allocator

Signed-off-by: Aleksandr Ivanov <[email protected]>

alexander-e1off added 30 commits November 26, 2024 15:42

Add seqnum and offset input args, add composite sequence number support

4438797

Signed-off-by: Aleksandr Ivanov <[email protected]>

Debug binary search for all range types

465076c

Signed-off-by: Aleksandr Ivanov <[email protected]>

Reemove debug

32f36b3

Signed-off-by: Aleksandr Ivanov <[email protected]>

Simplify getValue template

e0cf5a1

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add search result decorators for seq.number and offset, add UTs

bd9001d

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix hasCache() call

cff0b26

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix code style

1fd8eac

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix code style

81ba90f

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add search for specific sequence numbers and offsets

8a5d825

Signed-off-by: Aleksandr Ivanov <[email protected]>

Cleanup

fd3f96b

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix README.md

42e6b9a

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix typo in README.md

2343748

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update header caption

5189f9c

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix review comments

80c58ce

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix code style

9175303

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add support of queueOp and journalOp records

ec47845

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add UTs, update footer printing

8a46f56

Signed-off-by: Aleksandr Ivanov <[email protected]>

Move printRecord methods to RecordPrinter namespace

d270645

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add printRecord method for queueOp record

98655ac

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add flag to stop search when reached higher bound

3e9f3f4

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update README.md

9be7be8

Signed-off-by: Aleksandr Ivanov <[email protected]>

Revert "Add flag to stop search when reached higher bound"

13522a6

Signed-off-by: Aleksandr Ivanov <[email protected]>

Add flag to stop search when reached higher bound

096a3fa

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update README.md, fix code style

77a57a5

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix typo

062979d

Signed-off-by: Aleksandr Ivanov <[email protected]>

Cosmetic change: move validators to CommandLineArguments

05192f0

Signed-off-by: Aleksandr Ivanov <[email protected]>

Cleanup

a82b95f

Signed-off-by: Aleksandr Ivanov <[email protected]>

Update README.md

d69dbf6

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix comment

15cc481

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix typo

042b1ff

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix delimiter

ab2fbba

Signed-off-by: Aleksandr Ivanov <[email protected]>

alexander-e1off marked this pull request as ready for review March 6, 2025 11:00

alexander-e1off requested a review from a team as a code owner March 6, 2025 11:00

alexander-e1off requested a review from 678098 March 6, 2025 11:01

alexander-e1off assigned 678098 Mar 6, 2025

alexander-e1off added 2 commits March 6, 2025 14:59

Update readme.md with missed json format

99288c3

Signed-off-by: Aleksandr Ivanov <[email protected]>

Fix validation

45001cd

Signed-off-by: Aleksandr Ivanov <[email protected]>