Add --check-lcm mode to verify captured LedgerCloseMeta golden data in CI

CONTRIBUTING.md

@@ -294,6 +294,50 @@ files. You should then inspect to see that only the transactions you expected to
 see change did so. If so, commit the changes as a new set of baselines for
 future tests.
 
+## Running and updating LedgerCloseMeta checks
+
+Alongside the TxMeta hashes above, the unit tests can capture the full
+`LedgerCloseMeta` XDR of every ledger they close, and check it against golden
+data stored in the repository. Where the TxMeta baselines record a hash per
+transaction, this records the complete binary meta, so the files also serve as
+test data for downstream consumers such as Horizon and RPC. The two modes are:
+
+  * `--capture-lcm` which writes golden files, and
+  * `--check-lcm <dirname>` which checks against them, where `<dirname>` is the
+    directory holding the golden trees (the source tree root).
+
+The golden data lives in `test-lcm-current` (for the current protocol) and
+`test-lcm-next` (for the next protocol), one subdirectory per test file, one
+file per leaf section, named by a truncated hash of the test and section names.
+Each subdirectory's `index.json` maps those hashes back to readable names and
+records the protocol version and rng seed that produced the data. Continuous
+integration runs `--check-lcm`, which fails fast if those headers do not match
+the running binary — so a protocol version bump requires re-capturing the
+data, even when no transaction semantics changed.
+
+To re-capture after an intentional change, or after a protocol bump:
+
+    stellar-core test [tx] --rng-seed 12345 --capture-lcm --prune-stale-lcm
+
+for a build with only the current protocol enabled, and the same command for a
+build configured with `--enable-next-protocol-version-unsafe-for-production`,
+which writes the `next` tier instead. Note there is no `--all-versions` here:
+the golden data is captured at the default (latest) protocol version only.
+
+`--prune-stale-lcm` deletes golden files the run did not write and rebuilds
+each `index.json` from what it captured, so leaves that are no longer produced
+do not accumulate. Pass it only with the full `[tx]` run above: a narrower run
+visits only some leaves, and pruning would delete golden data that is still
+valid.
+
+Some tests are automatically skipped, with the reason logged, because their
+meta cannot serve as golden data: those that inject ledger entries straight
+into the bucket list (the meta never shows the entries being created), those
+that run a multi-node `Simulation`, and those using a config whose ledger
+content depends on thread scheduling or randomized nomination. If you add a
+test that needs a golden vector, close its ledgers through the ordinary
+single-node path.
+
 ## Fastdev and non-unified Rust builds
 
 As of protocol 20, some components of stellar-core are written in Rust (notably

ci-build.sh

@@ -217,6 +217,15 @@ export STELLAR_CORE_TEST_PARAMS="--ll fatal -r simple --disable-dots --all-versi
 export SKIP_SOROBAN_TESTS=true
 time make check
 
+# Reuse the same build to check the captured LedgerCloseMeta golden data
+# under test-lcm-current/ or test-lcm-next/ (the binary picks its tier).
+# No --all-versions: the golden data is captured at the default (latest)
+# protocol version only.
+echo Running fixed check-lcm tests
+export TEST_SPEC='[tx]'
+export STELLAR_CORE_TEST_PARAMS="--ll fatal -r simple --disable-dots --rng-seed 12345 --check-lcm ${SRC_DIR}"
+time make check
+
 echo All done
 date
 

docs/software/commands.md

@@ -232,8 +232,37 @@ Command options can only by placed after command.
       directory maps hashes back to human-readable names. Each file contains
       stream-framed `LedgerCloseMeta` entries that can be decoded with
       `stellar-xdr decode --type LedgerCloseMeta --input stream-framed`.
-      Meta is normalized (sorted) before writing so that output is
-      deterministic given a fixed `--rng-seed`.
+      Non-deterministic diagnostic events are zeroed before writing, but
+      entries are otherwise written in their original order, which some
+      downstream consumers depend on. Comparisons — both the skip-rewrite
+      check during capture and `--check-lcm` — are done on normalized
+      (sorted) copies, so given a fixed `--rng-seed` files are only
+      rewritten or flagged on semantic changes. Each `index.json` is stamped
+      with the protocol version, rng seed and protocol-version list that
+      produced the data.
+      * `--prune-stale-lcm` : with `--capture-lcm`, after a clean run rewrite
+      each touched directory's `index.json` from what the run captured and
+      delete `.xdr` files it did not write. Use it for a full-corpus
+      regeneration; without it, leaves that are no longer produced keep their
+      files and index entries. It is opt-in because a run filtered to a
+      subset of tests visits only some leaves, so pruning would delete golden
+      data that is still valid.
+      * Tests whose `LedgerCloseMeta` cannot serve as golden data are skipped
+      automatically, with the reason logged: those that inject ledger entries
+      straight into the bucket list (the meta never shows the entries being
+      created), run a multi-node `Simulation`, or use a config whose ledger
+      content depends on thread scheduling or randomized nomination.
+      `--check-lcm` fails if golden data still exists for such a test.
+      * `--check-lcm <DIRNAME>` : check `LedgerCloseMeta` captured from tests
+      against the golden files under `DIRNAME/test-lcm-current/` (or
+      `test-lcm-next/` for vnext builds), where `DIRNAME` is the directory
+      containing the two trees (typically the source tree root). Fails fast
+      if the `index.json` headers don't match the running binary — e.g. after
+      a protocol version bump without regenerating the golden data — and
+      fails at the end of the run if any captured meta differs from the
+      corresponding golden file. Continuous integration runs this mode; after
+      intentional changes, regenerate with `--capture-lcm` under each build
+      configuration and commit the result.
   * The network passphrase is set to `(V) (;,,;) (V)` for all captured meta.
   * For [further info](https://github.com/philsquared/Catch/blob/master/docs/command-line.md)
     on possible options for test.
@@ -244,7 +273,7 @@ Command options can only by placed after command.
     generated by running the capture under each build configuration (the
     `next` tier requires a binary built with
     `--enable-next-protocol-version-unsafe-for-production`):
-    `stellar-core test --rng-seed 12345 '[tx]' --capture-lcm`
+    `stellar-core test --rng-seed 12345 '[tx]' --capture-lcm --prune-stale-lcm`
 * **upgrade-db**: Upgrades local database to current schema version. This is
   usually done automatically during stellar-core run or other command.
 * **verify-checkpoints**: Listens to the network until it observes a consensus

src/bucket/test/BucketTestUtils.h

@@ -5,6 +5,7 @@
 #pragma once
 
 #include "test/TestUtils.h"
+#include "test/test.h"
 #include "xdr/Stellar-ledger.h"
 
 namespace stellar
@@ -85,6 +86,10 @@ class LedgerManagerForBucketTests : public LedgerManagerImpl
         std::vector<LedgerKey> const& deadEntries,
         bool alsoAddActualEntries = false)
     {
+        // Entries injected straight into the bucket list appear in the ledger
+        // without any transaction producing them, so the resulting meta is
+        // not a faithful record of how the state came to be.
+        taintLcmCapture("injects ledger entries directly into the bucket list");
         mUseTestEntries = true;
         mAlsoAddActualEntries = alsoAddActualEntries;
         mTestInitEntries = initEntries;
@@ -97,6 +102,9 @@ class LedgerManagerForBucketTests : public LedgerManagerImpl
         std::vector<LedgerEntry> const& archiveEntries,
         std::vector<LedgerKey> const& restoredEntries)
     {
+        // As above: injected archive state has no transaction behind it, so
+        // the meta does not record how that state came to be.
+        taintLcmCapture("injects ledger entries directly into the bucket list");
         mUseTestEntries = true;
         mTestArchiveEntries = archiveEntries;
         mTestRestoredEntries = restoredEntries;

src/herder/test/HerderTests.cpp

@@ -8972,7 +8972,11 @@ TEST_CASE("network externalizes empty-tx-set on missing value", "[herder][tx]")
 
     REQUIRE(counter.count() > stopPoint);
 
-    // Capture meta for use with --capture-lcm
+    // Capture meta for use with --capture-lcm. Note that the LCM capture
+    // eligibility rules currently exclude this test (multi-node simulation
+    // plus randomized nomination), so no golden data is produced today; if
+    // the simulation is ever made deterministic the vector comes back for
+    // free.
     txtest::captureLastClosedLedgerLcm(*app);
 }
 

src/simulation/Simulation.cpp

@@ -41,6 +41,10 @@ Simulation::Simulation(Mode mode, Hash const& networkID, ConfigGen confGen,
     , mConfigGen(confGen)
     , mQuorumSetAdjuster(qSetAdjust)
 {
+    // A multi-node simulation's ledger progression depends on thread
+    // scheduling and on how many SCP rounds happen to run, so meta captured
+    // from one is not reproducible across platforms or runs.
+    taintLcmCapture("runs a multi-node Simulation");
     auto cfg = newConfig();
     auto& parallel = cfg.BACKGROUND_OVERLAY_PROCESSING;
     parallel = parallel && mVirtualClockMode == VirtualClock::REAL_TIME;

src/test/TxTests.cpp

@@ -73,6 +73,19 @@ captureLastClosedLedgerLcm(Application& app)
     // then applies the tx directly, so txs never appear in LCM. Fix this
     // by restructuring applyCheck to use closeLedger(app, {tx}) when
     // capturing LCM.
+    if (isLcmCaptureEnabled())
+    {
+        // Configs whose ledger content depends on thread scheduling or on
+        // randomized nomination cannot produce reproducible golden data.
+        if (app.getConfig().EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD)
+        {
+            taintLcmCapture("uses EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD");
+        }
+        if (app.getConfig().TESTING_NOMINATE_RANDOM_VALUES)
+        {
+            taintLcmCapture("uses TESTING_NOMINATE_RANDOM_VALUES");
+        }
+    }
     if (isLcmCaptureEnabled() && !app.getConfig().MODE_USES_IN_MEMORY_LEDGER)
     {
         auto const& closeMeta =

src/test/TxTests.h

@@ -381,7 +381,8 @@ bool isSuccessResult(TransactionResult const& res);
 TestAccount getGenesisAccount(Application& app, uint32_t accountIndex);
 
 // Accumulated LedgerCloseMeta from closeLedger/closeLedgerOn calls.
-// Only accumulates when --capture-lcm is passed on the test command line.
+// Only accumulates when --capture-lcm or --check-lcm is passed on the test
+// command line.
 // Note: LCM capture does not cover BucketTestUtils::closeLedger or direct
 // externalizeValue calls (e.g. genesis ledger from app->start()).
 std::vector<LedgerCloseMeta> const& getAccumulatedLcm();