Feat[Storagetool]: cluster state ledger (CSL) file search support

src/applications/bmqstoragetool/README.md

@@ -2,22 +2,30 @@ BMQStorageTool
 ==============
 
 BMQStorageTool is a command-line tool for analyzing of BlazingMQ Broker storage
-files. It allows to search records in `journal` file with set of different 
-filters and output found results in the short or detail form.
-As input, a `journal` file (*.bmq_journal) is *always* required. To dump 
-payload, `data` file (*.bmq_data) is required. To filter by queue Uri, cluster
-state ledger (CSL) file (*.bmq_csl) is required.
+files. It allows to search records in:
+- `journal` file (.bmq_journal) with the set of different filters and output found results in 
+    the short or detail form;
+- `cluster state ledger` (CSL) file (*.bmq_csl) with the set of different filters and output 
+    found results in the short or detail form;  
+    NOTE: For this mode, `journal` file (.bmq_journal) **must not** be passed.
+
+As an input, either a `journal` file (*.bmq_journal) or `cluster state ledger` 
+(CSL) file (*.bmq_csl) is **always** required. To dump payload, `data` file (*.bmq_data) 
+is required. To filter by queue Uri, cluster state ledger (CSL) file (*.bmq_csl) is required.
 
 The tool can be found under your `CMAKE` build directory after making 
 the project. From the command-line, there are a few options you can use when
 invoking the tool.
 
 ```bash
 Usage:   bmqstoragetool [-r|record-type <record type>]*
+                        [--csl-record-type <csl record type>]*
                         [--journal-path <journal path>]
                         [--journal-file <journal file>]
                         [--data-file <data file>]
                         [--csl-file <csl file>]
+                        [--csl-from-begin]
+                        [--print-mode <print mode>]
                         [--guid <guid>]*
                         [--seqnum <seqnum>]*
                         [--offset <offset>]*
@@ -35,12 +43,15 @@ Usage:   bmqstoragetool [-r|record-type <record type>]*
                         [--details]
                         [--dump-payload]
                         [--dump-limit <dump limit>]
-                        [--min-records-per-queue <threshold>]
                         [--summary]
+                        [--min-records-per-queue <threshold>]
+                        [--summary-queues-limit <queues limit>]                        
                         [-h|help]
 Where:
   -r | --record-type          <record type>
-          record type to search {message|queue-op|journal-op} (default: message)
+          record type to search {<message>|queue-op|journal-op} (default: message)
+       --csl-record-type      <csl record type>
+          CSL record type to search {<snapshot>|update|commit|ack} (default: all record types)
        --journal-path         <pattern>
           '*'-ended file path pattern, where the tool will try to find journal
           and data files
@@ -50,6 +61,10 @@ Where:
           path to a .bmq_data file
        --csl-file             <csl file>
           path to a .bmq_csl file
+       --csl-from-begin
+          force to iterate CSL file from the beginning. By default: iterate from the latest snapshot
+       --print-mode           <print mode>
+          can be one of the following {<human>|json-prety|json-line} (default: human)
        --guid                 <guid>
           message guid
        --seqnum               <seqnum>
@@ -89,15 +104,17 @@ Where:
        --summary
           summary of all matching messages (number of outstanding messages and
           other statistics)
+       --summary-queues-limit   <queues limit>
+          limit of queues to display in CSL file summary (default: 50)
   -h | --help
           print usage
 ```
 
-Scenarios of BMQStorageTool usage
-=================================
+Scenarios of BMQStorageTool usage for journal file
+==================================================
 
 Output summary for journal file
-----------------------------------------
+-------------------------------
 Example:
 ```bash
 ./bmqstoragetool.tsk --journal-file=<path> --summary
@@ -111,7 +128,7 @@ Example:
 ```
 
 Search and otput all queueOp/journalOp records or all records in journal file
---------------------------------------------------
+-----------------------------------------------------------------------------
 Example:
 ```bash
 ./bmqstoragetool.tsk --journal-file=<path> --record-type=queue-op
@@ -136,14 +153,48 @@ Example:
 ```
 
 Search all message GUIDs with payload dump in journal file
-----------------------------------------------------------------------
+----------------------------------------------------------
 Example:
 ```bash
 ./bmqstoragetool.tsk --journal-file=<journal-path> --data-file=<data-path> --dump-payload
 ./bmqstoragetool.tsk --journal-path=<path.*> --dump-payload
 ./bmqstoragetool.tsk --journal-path=<path.*> --dump-payload --dump-limit=64
 ```
 
+Scenarios of BMQStorageTool usage for CSL file
+==============================================
+
+Output summary for CSL file
+---------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --csl-file=<path> --summary
+./bmqstoragetool.tsk --csl-file=<path> --summary --summary-queues-limit=100
+```
+
+Output records from the beginning of CSL file 
+---------------------------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --csl-file=<path> --csl-from-begin
+```
+NOTE: by default search is done from the latest snapshot.
+
+Search and otput only desired record types in CSL file
+------------------------------------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --csl-file=<path> --csl-record-type=snapshot --csl-record-type=update
+```
+NOTE: `snapshot`, `update`, `commit` and `ack ` are supported. Without this option all record types are selected.
+
+Search and otput records details in CSL file
+--------------------------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --csl-file=<path> --details
+```
+
 Applying search filters to above scenarios
 ==========================================
 
@@ -153,6 +204,24 @@ Example:
 ```bash
 ./bmqstoragetool.tsk --journal-file=<path> --guid=<guid_1> --guid=<guid_N>
 ```
+NOTE: no other filters are allowed with this one. Not suitable for CSL file search.
+
+Filter messages with corresponding composite sequence numbers (defined in form `primaryLeaseId-sequenceNumber` for journal file or `electorTerm-sequenceNumber` for CSL file)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --journal-file=<path> --seqnum=<leaseId1-sequenceNumber_1> --seqnum=<leaseId_N-sequenceNumber_N>
+./bmqstoragetool.tsk --csl-file=<path> --seqnum=<electorTerm1-sequenceNumber_1> --seqnum=<electorTerm_N-sequenceNumber_N>
+```
+NOTE: no other filters are allowed with this one
+
+Filter messages with corresponding record offsets
+-------------------------------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --journal-file=<path> --offset=<offset_1> --offset=<offset_N>
+./bmqstoragetool.tsk --csl-file=<path> --offset=<offset_1> --offset=<offset_N>
+```
 NOTE: no other filters are allowed with this one
 
 Filter messages with corresponding composite sequence numbers (defined in form <primaryLeaseId-sequenceNumber>)
@@ -178,6 +247,27 @@ Example:
 ./bmqstoragetool.tsk --journal-file=<path> --timestamp-lt=<stamp>
 ./bmqstoragetool.tsk --journal-file=<path> --timestamp-gt=<stamp>
 ./bmqstoragetool.tsk --journal-file=<path> --timestamp-lt=<stamp1> --timestamp-gt=<stamp2>
+./bmqstoragetool.tsk --csl-file=<path> --timestamp-lt=<stamp1> --timestamp-gt=<stamp2>
+```
+
+Filter messages within composite sequence numbers (primaryLeaseId/electorTerm, sequenceNumber) range
+----------------------------------------------------------------------------------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --journal-file=<path> --seqnum-lt=<leaseId-sequenceNumber>
+./bmqstoragetool.tsk --journal-file=<path> --seqnum-gt=<leaseId-sequenceNumber>
+./bmqstoragetool.tsk --journal-file=<path> --seqnum-lt=<leaseId1-sequenceNumber1> --seqnum-gt=<leaseId2-sequenceNumber2>
+./bmqstoragetool.tsk --csl-file=<path> --seqnum-lt=<electorTerm1-sequenceNumber1> --seqnum-gt=<electorTerm2-sequenceNumber2>
+```
+
+Filter messages within record offsets range
+-------------------------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --journal-file=<path> --offset-lt=<offset>
+./bmqstoragetool.tsk --journal-file=<path> --offset-gt=<offset>
+./bmqstoragetool.tsk --journal-file=<path> --offset-lt=<offset1> --offset-gt=<offset2>
+./bmqstoragetool.tsk --csl-file=<path> --offset-lt=<offset1> --offset-gt=<offset2>
 ```
 
 Filter messages within composite sequence numbers (primaryLeaseId, sequenceNumber) range
@@ -203,16 +293,35 @@ Filter messages by queue key
 Example:
 ```bash
 ./bmqstoragetool.tsk --journal-file=<path> --queue-key=<key_1> --queue-key=<key_N>
+./bmqstoragetool.tsk --csl-file=<path> --queue-key=<key_1> --queue-key=<key_N>
 ```
 
 Filter messages by queue Uri
 ----------------------------
 Example:
 ```bash
 ./bmqstoragetool.tsk --journal-file=<journal_path> --csl-file=<csl_path> --queue-name=<queue_uri_1> --queue-name=<queue_uri_N>
+./bmqstoragetool.tsk --csl-file=<csl_path> --queue-name=<queue_uri_1> --queue-name=<queue_uri_N>
 ```
 NOTE: CSL file is required
 
+Output search results in machine readable (JSON) format for all above scenarios
+===============================================================================
+
+Output in JSON pretty format 
+----------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --print-mode=json-pretty
+```
+
+Output in JSON line format 
+--------------------------
+Example:
+```bash
+./bmqstoragetool.tsk --print-mode=json-line
+```
+
 Display number of records per type (e.g. Message, Confirm, Delete, etc.) per queue.
 The number of Confirm records are displayed per AppId if there are more than 1 AppId.
 The information is displayed for the queues with a total number of records greater or

src/applications/bmqstoragetool/bmqstoragetool.m.cpp

@@ -38,16 +38,19 @@ static bool parseArgs(CommandLineArguments& arguments,
 {
     bool showHelp = false;
 
-    bsl::vector<bsl::string> defaultRecordType(allocator);
-    defaultRecordType.push_back(CommandLineArguments::k_MESSAGE_TYPE);
-
     balcl::OptionInfo specTable[] = {
         {"r|record-type",
          "record type",
-         "record type to search {message|queue-op|journal-op}",
+         "record type to search {<message>|queue-op|journal-op}",
          balcl::TypeInfo(&arguments.d_recordType,
                          CommandLineArguments::isValidRecordType),
-         balcl::OccurrenceInfo(defaultRecordType)},
+         balcl::OccurrenceInfo::e_OPTIONAL},
+        {"csl-record-type",
+         "csl record type",
+         "CSL record type to search {<snapshot>|update|commit|ack}",
+         balcl::TypeInfo(&arguments.d_cslRecordType,
+                         CommandLineArguments::isValidCslRecordType),
+         balcl::OccurrenceInfo::e_OPTIONAL},
         {"journal-path",
          "journal path",
          "'*'-ended file path pattern, where the tool will try to find "
@@ -72,6 +75,12 @@ static bool parseArgs(CommandLineArguments& arguments,
          balcl::TypeInfo(&arguments.d_cslFile,
                          CommandLineArguments::isValidFileName),
          balcl::OccurrenceInfo::e_OPTIONAL},
+        {"csl-from-begin",
+         "start from begin",
+         "force to iterate CSL file from the beginning. By default: iterate "
+         "from the latest snapshot",
+         balcl::TypeInfo(&arguments.d_cslFromBegin),
+         balcl::OccurrenceInfo::e_OPTIONAL},
         {"print-mode",
          "print mode",
          "can be one of the following {human|json-pretty|json-line}",
@@ -175,6 +184,11 @@ static bool parseArgs(CommandLineArguments& arguments,
          "min number of records per queue for detailed info to be displayed",
          balcl::TypeInfo(&arguments.d_minRecordsPerQueue),
          balcl::OccurrenceInfo(0LL)},
+        {"summary-queues-limit",
+         "queues limit",
+         "limit of queues to display in CSL file summary",
+         balcl::TypeInfo(&arguments.d_cslSummaryQueuesLimit),
+         balcl::OccurrenceInfo(50)},
         {"h|help",
          "help",
          "print usage)",
@@ -227,10 +241,11 @@ int main(int argc, const char* argv[])
         fileManager.load(new (*allocator)
                              FileManagerImpl(arguments.d_journalFile,
                                              arguments.d_dataFile,
+                                             arguments.d_cslFile,
+                                             arguments.d_cslFromBegin,
                                              allocator));
         if (!arguments.d_cslFile.empty()) {
-            parameters.d_queueMap =
-                FileManagerImpl::buildQueueMap(arguments.d_cslFile, allocator);
+            fileManager->fillQueueMapFromCslFile(&parameters.d_queueMap);
             parameters.validateQueueNames();
         }
     }

src/applications/bmqstoragetool/m_bmqstoragetool_commandprocessorfactory.cpp

@@ -15,6 +15,8 @@
 
 // bmqstoragetool
 #include <m_bmqstoragetool_commandprocessorfactory.h>
+#include <m_bmqstoragetool_cslfileprocessor.h>
+#include <m_bmqstoragetool_journalfileprocessor.h>
 #include <m_bmqstoragetool_searchresultfactory.h>
 
 namespace BloombergLP {
@@ -36,38 +38,58 @@ CommandProcessorFactory::createCommandProcessor(
 
     bslma::Allocator* alloc = bslma::Default::allocator(allocator);
 
-    // Create printer
-    bsl::shared_ptr<Printer> printer = createPrinter(params->d_printMode,
-                                                     ostream,
-                                                     allocator);
+    if (params->d_cslMode) {
+        // Create CSL printer
+        bsl::shared_ptr<CslPrinter> printer =
+            createCslPrinter(params->d_printMode, ostream, allocator);
 
-    // Create payload dumper
-    bslma::ManagedPtr<PayloadDumper> payloadDumper;
-    if (params->d_dumpPayload) {
-        payloadDumper.load(new (*alloc)
-                               PayloadDumper(ostream,
-                                             fileManager->dataFileIterator(),
-                                             params->d_dumpLimit,
-                                             alloc),
-                           alloc);
-    }
+        // Create CslSearchResult for given 'params'.
+        bsl::shared_ptr<CslSearchResult> cslSearchResult =
+            SearchResultFactory::createCslSearchResult(params, printer, alloc);
 
-    // Create searchResult for given 'params'.
-    bsl::shared_ptr<SearchResult> searchResult =
-        SearchResultFactory::createSearchResult(params,
-                                                fileManager,
-                                                printer,
-                                                payloadDumper,
-                                                alloc);
-    // Create commandProcessor.
-    bslma::ManagedPtr<CommandProcessor> commandProcessor(
-        new (*alloc) JournalFileProcessor(params,
+        // Create CslFileProcessor
+        return bslma::ManagedPtr<CommandProcessor>(
+            new (*alloc) CslFileProcessor(params,
                                           fileManager,
-                                          searchResult,
+                                          cslSearchResult,
                                           ostream,
                                           alloc),
-        alloc);
-    return commandProcessor;
+            alloc);  // RETURN
+    }
+    else {
+        // Create printer
+        bsl::shared_ptr<Printer> printer = createPrinter(params->d_printMode,
+                                                         ostream,
+                                                         allocator);
+
+        // Create payload dumper
+        bslma::ManagedPtr<PayloadDumper> payloadDumper;
+        if (params->d_dumpPayload) {
+            payloadDumper.load(
+                new (*alloc) PayloadDumper(ostream,
+                                           fileManager->dataFileIterator(),
+                                           params->d_dumpLimit,
+                                           alloc),
+                alloc);
+        }
+
+        // Create searchResult for given 'params'.
+        bsl::shared_ptr<SearchResult> searchResult =
+            SearchResultFactory::createSearchResult(params,
+                                                    fileManager,
+                                                    printer,
+                                                    payloadDumper,
+                                                    alloc);
+        // Create commandProcessor.
+        bslma::ManagedPtr<CommandProcessor> commandProcessor(
+            new (*alloc) JournalFileProcessor(params,
+                                              fileManager,
+                                              searchResult,
+                                              ostream,
+                                              alloc),
+            alloc);
+        return commandProcessor;
+    }
 }
 
 }  // close package namespace