Consensus phase entropy

[sublimator]

featureConsensusEntropy: Decentralized Secure Randomness

Adding randomness to deterministic consensus sounds simple until you try to do it without breaking safety. This PR implements Same-Ledger Usable Randomness: finalizing entropy after user intent is locked, but before normal execution in that same ledger.

🔎 Review Scope

• Blast radius (high level): consensus proposal encoding, establish sub-state logic, pseudo-tx injection, ledger apply ordering, transactor apply path, and hook-facing entropy APIs.
• Amendment gating: featureConsensusEntropy is DefaultNo; behavior is inert until enabled by amendment vote.
• Migration/upgrade: no migration required, no config changes required, and no manual database steps; amendment-gated behavior remains inert until vote-in.
• Test coverage highlights: ConsensusEntropy_test (Hook dice()/random(), fallback semantics) and ExtendedPosition_test (serialization compatibility and malformed wire cases).

🛠 How It Works (The Final Solution)

The architecture centers on converging on signed input sets rather than voting on a derived output hash. This ensures that every node can independently verify and reconstruct the final result.

1. Transport: Piggybacked Proposals

The ConsensusProposal wire format is extended via ExtendedPosition. Most entropy data (commitments and reveals) flows through existing proposal gossip with low incremental payload overhead on the fast path, while consensus latency cost comes from the added sub-state progression/timeouts.

• Equality Firewall: ExtendedPosition::operator== only compares the txSetHash. RNG sub-state differences never stall the core consensus on user transactions.

2. Pipelined Sub-states

RNG progression runs inside internal establish sub-states. These are checkpoints within the existing consensus cadence:

• ConvergingTx: Normal transaction convergence while harvesting entropy commitments.
• ConvergingCommit: Locking the commitSet once an 80% quorum of trusted proposers is reached.
• ConvergingReveal: Targets reveals from 100% of known committers, bounded by timeout/fallback paths (including the 1.5s reveal timeout) to preserve liveness.

3. SHAMap Union Convergence

Harvested commitments and reveals are stored in ephemeral, unbacked SHAMaps.

• Honest validators always agree on inclusion (every valid contribution belongs).
• Differences are reconciled via Union Merge (monotonic set growth).
• If a packet is dropped, the node uses the native InboundTransactions pipeline to fetch only the missing leaves from peers.

4. Synthetic Injection & Same-Ledger Execution

Once reveals are collected, the final entropy is computed deterministically (sha512Half(sorted_reveals)).

• Synthetic Transaction: Right before buildLCL (Ledger Construction), the node locally synthesizes a ttCONSENSUS_ENTROPY pseudo-transaction.
• Deterministic Ordering: This transaction is sorted to execute first, ensuring its entropy is available to every Hook and user transaction in the same block.
• Verification: Because the inputs were agreed upon in consensus, nodes synthesize identical transactions. Any local fault is caught by the validation phase (Example 5 intuition + Theorem 8 safety framing).

⚓ Hook API Integration

Provides two new deterministic WebAssembly APIs for Hook developers:

• dice(sides): Returns a fair integer from 0 to sides-1.
• random(write_ptr, write_len): Fills a buffer with cryptographically secure consensus-derived randomness.

🛡 Safety & Liveness

• Safety: RNG machinery resides entirely in the deliberation path. Safety remains anchored to the validation-phase quorum (Chase & MacBrough 2018, §4.1 / Theorem 8).
• Liveness: Entropy availability degrades deterministically to a zero-path under extreme stress (e.g., impossible quorums or timeouts), ensuring the ledger always closes.

🛠 Infrastructure & Support Logic

Several non-obvious plumbing changes were required to make the RNG pipeline robust and testable:

1. Fast Polling during RNG Transitions

To reduce the latency impact of the extra sub-states, the heartbeat timer accelerates to 250ms (tunable via XAHAU_RNG_POLL_MS) while in the RNG pipeline.
📍 src/ripple/app/misc/NetworkOPs.cpp:992-1005

 992     // Use faster polling during RNG sub-state transitions
 993     // to reduce latency of commit-reveal rounds.
 994     // Tunable via XAHAU_RNG_POLL_MS env var (default 250ms).
 995     if (mConsensus.inRngSubState())
 996     {
 997         static auto const rngPollMs = []() -> std::chrono::milliseconds {
 998             if (auto const* env = std::getenv("XAHAU_RNG_POLL_MS"))
 999                 return std::chrono::milliseconds{std::atoi(env)};
1000             return std::chrono::milliseconds{250};
1001         }();
1002         setHeartbeatTimer(rngPollMs);
1003     }
1004     else
1005         setHeartbeatTimer();

2. Local Testnet Resource Charging

Connections from 127.0.0.1 normally share a single IP resource bucket. This change preserves the port for loopback addresses so that local multi-node testnets don't hit peer resource limits due to the increased RNG set traffic.
📍 src/ripple/resource/impl/Logic.h:113-117

 113         // Inbound connections from the same IP normally share one
 114         // resource bucket (port stripped) for DoS protection.  For
 115         // loopback addresses, preserve the port so local testnet nodes
 116         // each get their own bucket instead of all sharing one.
 117         auto const key = is_loopback(address) ? address : address.at_port(0);

3. Test Environment Gating

featureConsensusEntropy is excluded from default jtx::Env tests to prevent its automatic pseudo-tx injection from breaking existing test suites that rely on specific transaction counts.
📍 src/test/jtx/Env.h:86-89

  86         // TODO: ConsensusEntropy injects a pseudo-tx every ledger which
  87         // breaks existing test transaction count assumptions. Exclude from
  88         // default test set until dedicated tests are written.
  89         return FeatureBitset(feats) - featureConsensusEntropy;

4. Pseudo-transaction Filtering

Internal metadata (commits/reveals) is stored as pseudo-transactions in ephemeral SHAMaps for transport. This logic ensures they are filtered out and never submitted to the actual transaction processing engine.
📍 src/ripple/app/ledger/ConsensusTransSetSF.cpp:67-71

  67             // Don't submit pseudo-transactions (consensus entropy, fees,
  68             // amendments, etc.) — they exist as SHAMap entries for
  69             // content-addressed identification but are not real user txns.
  70             if (isPseudoTx(*stx))
  71                 return;

---

Guided Code Review (Projected Source)

This section follows runtime order so the code reads as a story, not a file dump.

1) Proposal payload: ExtendedPosition carries RNG sidecar fields

ExtendedPosition adds commit/reveal set identities and per-validator leaves while keeping tx-set identity explicit.
Non-obvious: operator== compares only txSetHash on purpose. That decouples core tx-set convergence from RNG sub-state drift, so RNG disagreements cannot deadlock transaction consensus.
operator== (equality firewall):
📍 src/ripple/app/consensus/RCLCxPeerPos.h:109-144

 109     bool
 110     operator==(ExtendedPosition const& other) const
 111     {
 112         return txSetHash == other.txSetHash;
 113     }
 114 
 115     bool
 116     operator!=(ExtendedPosition const& other) const
 117     {
 118         return !(*this == other);
 119     }
 120 
 121     // Comparison with uint256 (compares txSetHash only)
 122     bool
 123     operator==(uint256 const& hash) const
 124     {
 125         return txSetHash == hash;
 126     }
 127 
 128     bool
 129     operator!=(uint256 const& hash) const
 130     {
 131         return txSetHash != hash;
 132     }
 133 
 134     friend bool
 135     operator==(uint256 const& hash, ExtendedPosition const& pos)
 136     {
 137         return pos.txSetHash == hash;
 138     }
 139 
 140     friend bool
 141     operator!=(uint256 const& hash, ExtendedPosition const& pos)
 142     {
 143         return pos.txSetHash != hash;
 144     }

add() (signed serialization of all sidecar fields):
📍 src/ripple/app/consensus/RCLCxPeerPos.h:149-178

 149     void
 150     add(Serializer& s) const
 151     {
 152         s.addBitString(txSetHash);
 153 
 154         // Wire compatibility: if no extensions, emit exactly 32 bytes
 155         // so legacy nodes that expect a plain uint256 work unchanged.
 156         if (!commitSetHash && !entropySetHash && !myCommitment && !myReveal)
 157             return;
 158 
 159         std::uint8_t flags = 0;
 160         if (commitSetHash)
 161             flags |= 0x01;
 162         if (entropySetHash)
 163             flags |= 0x02;
 164         if (myCommitment)
 165             flags |= 0x04;
 166         if (myReveal)
 167             flags |= 0x08;
 168         s.add8(flags);
 169 
 170         if (commitSetHash)
 171             s.addBitString(*commitSetHash);
 172         if (entropySetHash)
 173             s.addBitString(*entropySetHash);
 174         if (myCommitment)
 175             s.addBitString(*myCommitment);
 176         if (myReveal)
 177             s.addBitString(*myReveal);
 178     }

fromSerialIter() (legacy + extended wire decode):
📍 src/ripple/app/consensus/RCLCxPeerPos.h:199-242

 199     static std::optional<ExtendedPosition>
 200     fromSerialIter(SerialIter& sit, std::size_t totalSize)
 201     {
 202         if (totalSize < 32)
 203             return std::nullopt;
 204 
 205         ExtendedPosition pos;
 206         pos.txSetHash = sit.get256();
 207 
 208         // Legacy format: exactly 32 bytes
 209         if (totalSize == 32)
 210             return pos;
 211 
 212         // Extended format: flags byte + optional uint256 fields
 213         if (sit.empty())
 214             return pos;
 215 
 216         std::uint8_t flags = sit.get8();
 217 
 218         // Reject unknown flag bits (reduces wire malleability)
 219         if (flags & 0xF0)
 220             return std::nullopt;
 221 
 222         // Validate exact byte count for the flagged fields.
 223         // Each flag bit indicates a 32-byte uint256.
 224         int fieldCount = 0;
 225         for (int i = 0; i < 4; ++i)
 226             if (flags & (1 << i))
 227                 ++fieldCount;
 228 
 229         if (sit.getBytesLeft() != static_cast<std::size_t>(fieldCount * 32))
 230             return std::nullopt;
 231 
 232         if (flags & 0x01)
 233             pos.commitSetHash = sit.get256();
 234         if (flags & 0x02)
 235             pos.entropySetHash = sit.get256();
 236         if (flags & 0x04)
 237             pos.myCommitment = sit.get256();
 238         if (flags & 0x08)
 239             pos.myReveal = sit.get256();
 240 
 241         return pos;
 242     }

2) Harvest stage: trust boundary + reveal verification

Incoming RNG data is rejected for non-UNL senders and reveals are accepted only if they match prior commitments.
Non-obvious: this is where "no-commit, no-reveal" is enforced. A reveal without a recorded commitment is dropped to block late reveal grinding.
📍 src/ripple/app/consensus/RCLConsensus.cpp:1791-1866

1791     // Reject data from validators not in the active UNL
1792     if (!isUNLReportMember(nodeId))
1793     {
1794         JLOG(j_.debug()) << "RNG: rejecting data from non-UNL validator "
1795                          << nodeId;
1796         return;
1797     }
1798 
1799     // Store nodeId -> publicKey mapping for deterministic ordering
1800     nodeIdToKey_[nodeId] = publicKey;
1801 
1802     // Harvest commitment if present
1803     if (position.myCommitment)
1804     {
1805         auto [it, inserted] =
1806             pendingCommits_.emplace(nodeId, *position.myCommitment);
1807         if (!inserted && it->second != *position.myCommitment)
1808         {
1809             JLOG(j_.warn())
1810                 << "Validator " << nodeId << " changed commitment from "
1811                 << it->second << " to " << *position.myCommitment;
1812             it->second = *position.myCommitment;
1813         }
1814         else if (inserted)
1815         {
1816             JLOG(j_.trace()) << "Harvested commitment from " << nodeId << ": "
1817                              << *position.myCommitment;
1818         }
1819     }
1820 
1821     // Harvest reveal if present — verify it matches the stored commitment
1822     if (position.myReveal)
1823     {
1824         auto commitIt = pendingCommits_.find(nodeId);
1825         if (commitIt == pendingCommits_.end())
1826         {
1827             // No commitment on record — cannot verify. Ignore to prevent
1828             // grinding attacks where a validator skips the commit phase.
1829             JLOG(j_.warn()) << "RNG: rejecting reveal from " << nodeId
1830                             << " (no commitment on record)";
1831             return;
1832         }
1833 
1834         // Verify Hash(reveal | pubKey | seq) == commitment
1835         auto const prevLgr = ledgerMaster_.getLedgerByHash(prevLedger);
1836         if (!prevLgr)
1837         {
1838             JLOG(j_.warn()) << "RNG: cannot verify reveal from " << nodeId
1839                             << " (prevLedger not available)";
1840             return;
1841         }
1842 
1843         auto const seq = prevLgr->info().seq + 1;
1844         auto const calculated = sha512Half(*position.myReveal, publicKey, seq);
1845 
1846         if (calculated != commitIt->second)
1847         {
1848             JLOG(j_.warn()) << "RNG: fraudulent reveal from " << nodeId
1849                             << " (does not match commitment)";
1850             return;
1851         }
1852 
1853         auto [it, inserted] =
1854             pendingReveals_.emplace(nodeId, *position.myReveal);
1855         if (!inserted && it->second != *position.myReveal)
1856         {
1857             JLOG(j_.warn()) << "Validator " << nodeId << " changed reveal from "
1858                             << it->second << " to " << *position.myReveal;
1859             it->second = *position.myReveal;
1860         }
1861         else if (inserted)
1862         {
1863             JLOG(j_.trace()) << "Harvested reveal from " << nodeId << ": "
1864                              << *position.myReveal;
1865         }
1866     }

3) Quorum basis: expected proposers first, UNL fallback

This quorum helper is the bridge between ideal participation and real network conditions.
Non-obvious: quorum is not a blind static UNL count in steady-state; expected proposers drive the fast path and UNL membership is the safety fallback.
📍 src/ripple/app/consensus/RCLConsensus.cpp:1154-1169

1154 std::size_t
1155 RCLConsensus::Adaptor::quorumThreshold() const
1156 {
1157     // Prefer expected proposers (recent proposers ∩ UNL) — this
1158     // adapts to actual network conditions rather than relying on
1159     // the potentially stale UNL Report.  Falls back to full
1160     // UNL Report for cold boot (first round).
1161     //
1162     // Round 1: threshold based on full UNL (conservative)
1163     // Round 2+: threshold based on who actually proposed last round
1164     auto const base = expectedProposers_.empty() ? unlReportNodeIds_.size()
1165                                                  : expectedProposers_.size();
1166     if (base == 0)
1167         return 1;  // safety: need at least one commit
1168     return (base * 80 + 99) / 100;
1169 }

4) State-machine checkpoints: ConvergingTx -> ConvergingCommit -> ConvergingReveal

This is the core sub-state progression inside establish; it gates commit quorum, reveal publication, timeout fallback, and entropy-set readiness.
Non-obvious: this is where liveness bounds are enforced (impossible quorum, pipeline timeout, reveal timeout) without stalling tx-set consensus, including the "timeout but still quorum => partial commit-set proceed" path.
📍 src/ripple/consensus/Consensus.h:1413-1593

1413     // --- RNG Sub-state Checkpoints (if adaptor supports RNG) ---
1414     // These sub-states use union convergence (not avalanche).
1415     // Commits and reveals arrive piggybacked on proposals, so by the time
1416     // we reach these checkpoints most data is already collected. The
1417     // SHAMap fetch/diff/merge in handleAcquiredRngSet is a safety net
1418     // for stragglers, not a voting mechanism.
1419     //
1420     // Why 80% for commits but 100% for reveals?
1421     //
1422     // COMMITS: quorum is based on the active UNL, but we don't know
1423     // which UNL members are actually online until they propose — and
1424     // commitments ride on those same proposals.  Chicken-and-egg: we
1425     // learn who's active by receiving their commits.  80% of the UNL
1426     // says "we've heard from enough validators, let's go."  The
1427     // impossible-quorum early-exit handles the case where too few
1428     // participants exist to ever reach 80%.
1429     //
1430     // REVEALS: the commit set is now locked and we know *exactly* who
1431     // committed.  Every committer broadcasts their reveal immediately.
1432     // So we wait for ALL of them, with rngREVEAL_TIMEOUT (measured
1433     // from ConvergingReveal entry) as the safety valve for nodes that
1434     // crash between commit and reveal.
1435     if constexpr (requires(Adaptor & a) {
1436                       a.hasQuorumOfCommits();
1437                       a.buildCommitSet(typename Ledger_t::Seq{});
1438                       a.generateEntropySecret();
1439                   })
1440     {
1441         auto const buildSeq = previousLedger_.seq() + typename Ledger_t::Seq{1};
1442 
1443         JLOG(j_.debug()) << "RNG: phaseEstablish estState="
1444                          << static_cast<int>(estState_);
1445 
1446         if (estState_ == EstablishState::ConvergingTx)
1447         {
1448             if (adaptor_.hasQuorumOfCommits())  // all expected proposers (80%
1449                                                 // fallback)
1450             {
1451                 auto commitSetHash = adaptor_.buildCommitSet(buildSeq);
1452 
1453                 // Keep the same entropy secret from onClose() — do NOT
1454                 // regenerate.  The commitment in the commitSet was built
1455                 // from that original secret; regenerating would make the
1456                 // later reveal fail verification.
1457                 auto newPos = result_->position.position();
1458                 newPos.commitSetHash = commitSetHash;
1459 
1460                 result_->position.changePosition(
1461                     newPos, asCloseTime(result_->position.closeTime()), now_);
1462 
1463                 if (mode_.get() == ConsensusMode::proposing)
1464                     adaptor_.propose(result_->position);
1465 
1466                 estState_ = EstablishState::ConvergingCommit;
1467                 JLOG(j_.debug()) << "RNG: transitioned to ConvergingCommit"
1468                                  << " commitSet=" << commitSetHash;
1469                 return;  // Wait for next tick
1470             }
1471 
1472             // Don't let the round close while waiting for commit quorum.
1473             // Without this gate, execution falls through to the normal
1474             // consensus close logic and nodes inject partial/zero entropy
1475             // while others are still collecting — causing ledger mismatches.
1476             //
1477             // However, if we've already converged on the txSet (which we
1478             // have — haveConsensus() passed above) and there aren't enough
1479             // participants to ever reach quorum, skip immediately.  With
1480             // 3 nodes and quorum=3, losing one node means 2/3 commits
1481             // forever — waiting 3s per round just delays recovery.
1482             //
1483             // NOTE: Late-joining nodes (e.g. restarting after a crash)
1484             // cannot help here.  They enter the round as proposing=false
1485             // and onClose() skips commitment generation for non-proposers.
1486             // It takes at least one full round of observing before
1487             // consensus promotes them to proposing.
1488             {
1489                 // participants = peers + ourselves
1490                 auto const participants = currPeerPositions_.size() + 1;
1491                 auto const threshold = adaptor_.quorumThreshold();
1492                 bool const impossible = participants < threshold;
1493 
1494                 if (impossible)
1495                 {
1496                     JLOG(j_.debug())
1497                         << "RNG: skipping commit wait (participants="
1498                         << participants << " < threshold=" << threshold << ")";
1499                     // Fall through to close with zero entropy
1500                 }
1501                 else
1502                 {
1503                     bool timeout =
1504                         result_->roundTime.read() > parms.rngPIPELINE_TIMEOUT;
1505                     if (!timeout)
1506                         return;  // Wait for more commits
1507 
1508                     // Timeout waiting for all expected proposers.
1509                     // If we still have quorum (80% of UNL), proceed
1510                     // with what we have — the SHAMap merge handles
1511                     // any fuzziness for this transition round.
1512                     auto const commits = adaptor_.pendingCommitCount();
1513                     auto const quorum = adaptor_.quorumThreshold();
1514                     if (commits >= quorum)
1515                     {
1516                         JLOG(j_.info())
1517                             << "RNG: commit timeout but have quorum ("
1518                             << commits << "/" << quorum
1519                             << "), proceeding with partial set";
1520                         // Jump to the same path as hasQuorumOfCommits
1521                         auto commitSetHash = adaptor_.buildCommitSet(buildSeq);
1522                         auto newPos = result_->position.position();
1523                         newPos.commitSetHash = commitSetHash;
1524                         result_->position.changePosition(
1525                             newPos,
1526                             asCloseTime(result_->position.closeTime()),
1527                             now_);
1528                         if (mode_.get() == ConsensusMode::proposing)
1529                             adaptor_.propose(result_->position);
1530                         estState_ = EstablishState::ConvergingCommit;
1531                         JLOG(j_.debug())
1532                             << "RNG: transitioned to ConvergingCommit"
1533                             << " commitSet=" << commitSetHash
1534                             << " (timeout fallback)";
1535                         return;
1536                     }
1537                     // Truly below quorum: fall through to zero entropy
1538                 }
1539             }
1540         }
1541         else if (estState_ == EstablishState::ConvergingCommit)
1542         {
1543             // haveConsensus() implies agreement on commitSetHash
1544             auto newPos = result_->position.position();
1545             newPos.myReveal = adaptor_.getEntropySecret();
1546 
1547             result_->position.changePosition(
1548                 newPos, asCloseTime(result_->position.closeTime()), now_);
1549 
1550             if (mode_.get() == ConsensusMode::proposing)
1551                 adaptor_.propose(result_->position);
1552 
1553             estState_ = EstablishState::ConvergingReveal;
1554             revealPhaseStart_ = std::chrono::steady_clock::now();
1555             JLOG(j_.debug()) << "RNG: transitioned to ConvergingReveal"
1556                              << " reveal=" << adaptor_.getEntropySecret();
1557             return;  // Wait for next tick
1558         }
1559         else if (estState_ == EstablishState::ConvergingReveal)
1560         {
1561             // Wait for ALL committers to reveal (not just 80%).
1562             // Timeout measured from ConvergingReveal entry, not round start.
1563             auto const elapsed =
1564                 std::chrono::steady_clock::now() - revealPhaseStart_;
1565             bool timeout = elapsed > parms.rngREVEAL_TIMEOUT;
1566             bool ready = false;
1567 
1568             if ((haveConsensus() && adaptor_.hasMinimumReveals()) || timeout)
1569             {
1570                 if (timeout && !adaptor_.hasAnyReveals())
1571                 {
1572                     adaptor_.setEntropyFailed();
1573                     JLOG(j_.warn()) << "RNG: entropy failed (no reveals)";
1574                 }
1575                 else
1576                 {
1577                     auto entropySetHash = adaptor_.buildEntropySet(buildSeq);
1578                     auto newPos = result_->position.position();
1579                     newPos.entropySetHash = entropySetHash;
1580 
1581                     result_->position.changePosition(
1582                         newPos,
1583                         asCloseTime(result_->position.closeTime()),
1584                         now_);
1585                     JLOG(j_.debug()) << "RNG: built entropySet";
1586                 }
1587                 ready = true;
1588             }
1589 
1590             if (!ready)
1591                 return;
1592         }
1593     }

5) SHAMap construction: commit/reveal sets with proof blobs

These build the ephemeral unbacked SHAMaps and embed proposal proof blobs (sfBlob) used for verification on fetch/merge.
Non-obvious: commit-set and entropy-set proof sources are intentionally different (commitProofs_ for commit leaves, proposalProofs_ for reveal leaves) to keep set construction deterministic across nodes.
📍 src/ripple/app/consensus/RCLConsensus.cpp:1272-1323

1272     auto map =
1273         std::make_shared<SHAMap>(SHAMapType::TRANSACTION, app_.getNodeFamily());
1274     map->setUnbacked();
1275 
1276     // NOTE: avoid structured bindings in for-loops containing lambdas —
1277     // clang-14 (CI) rejects capturing them (P2036R3 not implemented).
1278     for (auto const& entry : pendingCommits_)
1279     {
1280         auto const& nid = entry.first;
1281         auto const& commit = entry.second;
1282 
1283         if (!isUNLReportMember(nid))
1284             continue;
1285 
1286         auto kit = nodeIdToKey_.find(nid);
1287         if (kit == nodeIdToKey_.end())
1288             continue;
1289 
1290         // Encode the NodeID into sfAccount so handleAcquiredRngSet can
1291         // recover it without recomputing (master vs signing key issue).
1292         AccountID acctId;
1293         std::memcpy(acctId.data(), nid.data(), acctId.size());
1294 
1295         STTx tx(ttCONSENSUS_ENTROPY, [&](auto& obj) {
1296             obj.setFieldU32(sfFlags, tfEntropyCommit);
1297             obj.setFieldU32(sfLedgerSequence, seq);
1298             obj.setAccountID(sfAccount, acctId);
1299             obj.setFieldU32(sfSequence, 0);
1300             obj.setFieldAmount(sfFee, STAmount{});
1301             obj.setFieldH256(sfDigest, commit);
1302             obj.setFieldVL(sfSigningPubKey, kit->second.slice());
1303             auto proofIt = commitProofs_.find(nid);
1304             if (proofIt != commitProofs_.end())
1305                 obj.setFieldVL(sfBlob, serializeProof(proofIt->second));
1306         });
1307 
1308         Serializer s(2048);
1309         tx.add(s);
1310         map->addItem(
1311             SHAMapNodeType::tnTRANSACTION_NM,
1312             make_shamapitem(tx.getTransactionID(), s.slice()));
1313     }
1314 
1315     map = map->snapShot(false);
1316     commitSetMap_ = map;
1317 
1318     auto const hash = map->getHash().as_uint256();
1319     inboundTransactions_.giveSet(hash, map, false);
1320 
1321     JLOG(j_.debug()) << "RNG: built commitSet SHAMap hash=" << hash
1322                      << " entries=" << pendingCommits_.size();
1323     return hash;

📍 src/ripple/app/consensus/RCLConsensus.cpp:1331-1379

1331     auto map =
1332         std::make_shared<SHAMap>(SHAMapType::TRANSACTION, app_.getNodeFamily());
1333     map->setUnbacked();
1334 
1335     // NOTE: avoid structured bindings — clang-14 can't capture them (P2036R3).
1336     for (auto const& entry : pendingReveals_)
1337     {
1338         auto const& nid = entry.first;
1339         auto const& reveal = entry.second;
1340 
1341         if (!isUNLReportMember(nid))
1342             continue;
1343 
1344         auto kit = nodeIdToKey_.find(nid);
1345         if (kit == nodeIdToKey_.end())
1346             continue;
1347 
1348         AccountID acctId;
1349         std::memcpy(acctId.data(), nid.data(), acctId.size());
1350 
1351         STTx tx(ttCONSENSUS_ENTROPY, [&](auto& obj) {
1352             obj.setFieldU32(sfFlags, tfEntropyReveal);
1353             obj.setFieldU32(sfLedgerSequence, seq);
1354             obj.setAccountID(sfAccount, acctId);
1355             obj.setFieldU32(sfSequence, 0);
1356             obj.setFieldAmount(sfFee, STAmount{});
1357             obj.setFieldH256(sfDigest, reveal);
1358             obj.setFieldVL(sfSigningPubKey, kit->second.slice());
1359             auto proofIt = proposalProofs_.find(nid);
1360             if (proofIt != proposalProofs_.end())
1361                 obj.setFieldVL(sfBlob, serializeProof(proofIt->second));
1362         });
1363 
1364         Serializer s(2048);
1365         tx.add(s);
1366         map->addItem(
1367             SHAMapNodeType::tnTRANSACTION_NM,
1368             make_shamapitem(tx.getTransactionID(), s.slice()));
1369     }
1370 
1371     map = map->snapShot(false);
1372     entropySetMap_ = map;
1373 
1374     auto const hash = map->getHash().as_uint256();
1375     inboundTransactions_.giveSet(hash, map, false);
1376 
1377     JLOG(j_.debug()) << "RNG: built entropySet SHAMap hash=" << hash
1378                      << " entries=" << pendingReveals_.size();
1379     return hash;

6) Injection stage (A): final entropy selection with deterministic fallback

Entropy is selected from verified reveals, with explicit standalone/zero fallback behavior for liveness.
Non-obvious: standalone mode deliberately uses synthetic deterministic entropy for local/dev usability; production safety semantics come from the non-standalone commit/reveal path. Zero entropy is the deterministic liveness fallback when entropy fails.
📍 src/ripple/app/consensus/RCLConsensus.cpp:1692-1745

1692     // Calculate entropy from collected reveals
1693     if (app_.config().standalone())
1694     {
1695         // Standalone mode: generate synthetic deterministic entropy
1696         // so that Hook APIs (dice/random) work for testing.
1697         finalEntropy = sha512Half(std::string("standalone-entropy"), seq);
1698         hasEntropy = true;
1699         JLOG(j_.info()) << "RNG: Standalone synthetic entropy " << finalEntropy
1700                         << " for ledger " << seq;
1701     }
1702     else if (entropyFailed_ || pendingReveals_.empty())
1703     {
1704         // Liveness fallback: inject zero entropy.
1705         // Hooks MUST check for zero to know entropy is unavailable.
1706         finalEntropy.zero();
1707         hasEntropy = true;
1708         JLOG(j_.warn()) << "RNG: Injecting ZERO entropy (fallback) for ledger "
1709                         << seq;
1710     }
1711     else
1712     {
1713         // Sort reveals deterministically by Validator Public Key
1714         std::vector<std::pair<PublicKey, uint256>> sorted;
1715         sorted.reserve(pendingReveals_.size());
1716 
1717         for (auto const& [nodeId, reveal] : pendingReveals_)
1718         {
1719             auto it = nodeIdToKey_.find(nodeId);
1720             if (it != nodeIdToKey_.end())
1721                 sorted.emplace_back(it->second, reveal);
1722         }
1723 
1724         if (!sorted.empty())
1725         {
1726             std::sort(
1727                 sorted.begin(), sorted.end(), [](auto const& a, auto const& b) {
1728                     return a.first.slice() < b.first.slice();
1729                 });
1730 
1731             // Mix all reveals into final entropy
1732             Serializer s;
1733             for (auto const& [key, reveal] : sorted)
1734             {
1735                 s.addVL(key.slice());
1736                 s.addBitString(reveal);
1737             }
1738             finalEntropy = sha512Half(s.slice());
1739             hasEntropy = true;
1740 
1741             JLOG(j_.info()) << "RNG: Injecting entropy " << finalEntropy
1742                             << " from " << sorted.size() << " reveals"
1743                             << " for ledger " << seq;
1744         }
1745     }

7) Injection stage (B): build and enqueue ttCONSENSUS_ENTROPY

The pseudo-tx is synthesized with deterministic fields and inserted into the canonical set.
Non-obvious: sfEntropyCount is part of the contract. It lets consumers distinguish healthy entropy contribution depth from fallback/low-participation rounds. Also, insertion into retriableTxs is legacy naming compatibility with the existing build/apply pipeline.
📍 src/ripple/app/consensus/RCLConsensus.cpp:1749-1769

1749     // Synthesize and inject the pseudo-transaction
1750     if (hasEntropy)
1751     {
1752         // Account Zero convention for pseudo-transactions (same as ttFEE, etc)
1753         auto const entropyCount = static_cast<std::uint16_t>(
1754             app_.config().standalone()
1755                 ? 20  // synthetic: high enough for Hook APIs (need >= 5)
1756                 : (entropyFailed_ || pendingReveals_.empty()
1757                        ? 0
1758                        : pendingReveals_.size()));
1759         STTx tx(ttCONSENSUS_ENTROPY, [&](auto& obj) {
1760             obj.setFieldU32(sfLedgerSequence, seq);
1761             obj.setAccountID(sfAccount, AccountID{});
1762             obj.setFieldU32(sfSequence, 0);
1763             obj.setFieldAmount(sfFee, STAmount{});
1764             obj.setFieldH256(sfDigest, finalEntropy);
1765             obj.setFieldU16(sfEntropyCount, entropyCount);
1766         });
1767 
1768         retriableTxs.insert(std::make_shared<STTx>(std::move(tx)));
1769     }

8) Build stage: entropy pseudo-tx executes before normal transactions

BuildLedger::applyTransactions applies ttCONSENSUS_ENTROPY first so in-ledger consumers can read it.
Non-obvious: this is the "same-ledger usability" guarantee point. If this ordering moves, dice()/random() semantics shift immediately.
📍 src/ripple/app/ledger/impl/BuildLedger.cpp:108-145

 108     // CRITICAL: Apply consensus entropy pseudo-tx FIRST before any other
 109     // transactions. This ensures hooks can read entropy during this ledger.
 110     for (auto it = txns.begin(); it != txns.end(); /* manual */)
 111     {
 112         if (it->second->getTxnType() != ttCONSENSUS_ENTROPY)
 113         {
 114             ++it;
 115             continue;
 116         }
 117 
 118         auto const txid = it->first.getTXID();
 119         JLOG(j.debug()) << "Applying entropy tx FIRST: " << txid;
 120 
 121         try
 122         {
 123             auto const result =
 124                 applyTransaction(app, view, *it->second, true, tapNONE, j);
 125 
 126             if (result == ApplyResult::Success)
 127             {
 128                 ++count;
 129                 JLOG(j.debug()) << "Entropy tx applied successfully";
 130             }
 131             else
 132             {
 133                 failed.insert(txid);
 134                 JLOG(j.warn()) << "Entropy tx failed to apply";
 135             }
 136         }
 137         catch (std::exception const& ex)
 138         {
 139             JLOG(j.warn()) << "Entropy tx throws: " << ex.what();
 140             failed.insert(txid);
 141         }
 142 
 143         it = txns.erase(it);
 144         break;  // Only one entropy tx per ledger
 145     }

9) Apply stage: write consensus entropy into the singleton ledger object

The transactor updates keylet::consensusEntropy() with digest/count/ledger-sequence deterministically.
Non-obvious: singleton-key write means there is exactly one consensus-entropy state target per ledger; deterministic write target is as important as deterministic value.
📍 src/ripple/app/tx/impl/Change.cpp:242-258

 242     auto sle = view().peek(keylet::consensusEntropy());
 243     bool const created = !sle;
 244 
 245     if (created)
 246         sle = std::make_shared<SLE>(keylet::consensusEntropy());
 247 
 248     sle->setFieldH256(sfDigest, entropy);
 249     sle->setFieldU16(sfEntropyCount, ctx_.tx.getFieldU16(sfEntropyCount));
 250     sle->setFieldU32(sfLedgerSequence, view().info().seq);
 251     // Note: sfPreviousTxnID and sfPreviousTxnLgrSeq are set automatically
 252     // by ApplyStateTable::threadItem() because isThreadedType() returns true
 253     // for ledger entries that have sfPreviousTxnID in their format.
 254 
 255     if (created)
 256         view().insert(sle);
 257     else
 258         view().update(sle);

10) Wire anchor: proposal message carrying extended payload bytes

TMProposeSet is the network envelope used for proposal payload transport.
Non-obvious: despite the legacy field name/comment, currenttxhash carries serialized proposal-position bytes (ExtendedPosition) in this path; backward compatibility is preserved because the legacy 32-byte txSet-only form remains valid.
📍 src/ripple/proto/ripple.proto:218-234

 218 message TMProposeSet
 219 {
 220     required uint32 proposeSeq          = 1;
 221     required bytes currentTxHash        = 2;    // the hash of the ledger we are proposing
 222     required bytes nodePubKey           = 3;
 223     required uint32 closeTime           = 4;
 224     required bytes signature            = 5;    // signature of above fields
 225     required bytes previousledger       = 6;
 226     repeated bytes addedTransactions    = 10;   // not required if number is large
 227     repeated bytes removedTransactions  = 11;   // not required if number is large
 228 
 229     // node vouches signature is correct
 230     optional bool checkedSignature      = 7     [deprecated=true];
 231 
 232     // Number of hops traveled
 233     optional uint32 hops                = 12    [deprecated=true];
 234 }

---

[Architectural Retrospective]

The Road to Consensus-Native Randomness: A Retrospective

A narrative history of how the RNG architecture evolved from early featureRNG experiments into the final featureConsensusEntropy design.

Adding randomness to deterministic consensus sounds simple until you try to do it without breaking safety.

Consensus requires determinism: every honest node must compute the same state transition.
Randomness requires unpredictability: nobody should know the final value early enough to game it.

The requirement that made this hard was not just "randomness," but same-ledger usable randomness: finalize entropy after user intent is locked, but before normal execution in that same ledger.

That path was not linear.

Part I: What the First Branch Taught Us (featRNG)

The initial branch was aggressively practical: reuse existing transaction paths, avoid deep consensus surgery, and move fast.

Experiment 1: ttRNG looked straightforward, then failed quickly

The earliest model used a single transaction path (ttRNG) with validator-generated entropy.

It failed for a concrete reason: entropy bytes entered open-ledger transaction flow too early.
That made the randomness path mempool-observable and timing-sensitive, so sophisticated actors could condition behavior around visible entropy before the round was fully sealed.

Very quickly, the branch moved toward a dual-model design (ttENTROPY + ttSHUFFLE) to try to close that timing gap.

Experiment 2: dual-model defense (ttENTROPY + ttSHUFFLE)

The next design split responsibilities:

• ttENTROPY: a UNL Validator Transaction (UVTx) — zero fee, seq=0, signed by the validator's ephemeral key, validated by UNLReport membership — used to submit blinded entropy hashes and later reveal them.
• ttSHUFFLE: a pseudo-transaction that derived extra entropy from proposal signatures, timed to land after the transaction set was frozen.

Conceptually, this was smart defense-in-depth. Operationally, it hit three structural problems:

• circular timing dependency around proposal-signature-derived shuffle values,
• overlap windows between OPEN and ESTABLISH creating nondeterministic inclusion,
• transaction-volume overhead (~70+ RNG-related tx artifacts per ledger, roughly ~28 GB/year of historical bloat) that became unacceptable at scale.

Experiment 3: mitigation hacks and why they still were not enough

Deterministic self-shuffle and piggyback variants improved specific failure modes. They did not remove the deeper issue: the model remained timing-sensitive and complex under real asynchronous behavior.

This was the "env var city" period (XAHAU_SELF_SHUFFLE, XAHAU_PIGGYBACK_SHUFFLE, and briefly XAHAU_AUTO_ACCEPT_SHUFFLES): useful for exposing failure boundaries, but also a clear signal that the architecture was being patched against the grain of consensus.

Experiment 4: dedicated shuffle phase (Open -> Establish -> Shuffle -> Validate)

The branch then tried full structural separation: a top-level shuffle phase, custom RNG message flow (TMRNGProposal), an RNGService managing commits/reveals in simple std::maps, and a forceRevealPhase() sync point to keep nodes aligned.

This delivered one lasting insight: contributors should be tied to actual recent consensus participants (the seed of later expected-proposer logic).

But the phase itself was abandoned:

• too much new state-machine surface area (new top-level phase + new protocol message = edge cases at every transition boundary),
• synchronization fragility (nodes drifted across the phase boundary under real latency, hitting the same timing problems the earlier model had — just at a different seam),
• no native SHAMap diff/fetch leverage (simple maps meant building custom retry/fetch logic for missed messages).

The conclusion from Part I was precise:
commit/reveal was the right cryptographic primitive, but the transport/convergence model was wrong.

The final commit on the initial branch landed one more practical insight: entropy participation should track actual establish-round participation (establishProposers), not just static UNL membership. That expected-participant logic survived into the final architecture even as the dedicated shuffle phase did not.

---

Part II: The Trap We Nearly Chose (Scalar Opinion Convergence)

The seductive simplification was to treat entropy like any disputed scalar:
"let nodes publish their computed entropy value and avalanche-converge on the majority."

A lightweight discrete-event simulator (sim/rng_sim.cpp) was built to pressure-test this assumption under realistic latency and packet asymmetry. (This was a quick prototype model, not a faithful rippled consensus simulator — but it was sufficient to expose the core pathology.)

This fails for a reason that became impossible to ignore:

• entropy output is subset-dependent,
• subset-dependent outputs do not cluster naturally,
• and non-clustered outputs give Avalanche no stable flipping signal.

When node A computes from set S_A and node B from S_B, and S_A != S_B, their scalars are unrelated.
At that point, you face a bad fork in design philosophy:

1. Blind adoption: flip to whatever value seems popular.
2. Principled reconciliation: fetch missing inputs, verify, recompute deterministically.

The final architecture deliberately chose option 2.

---

Part III: The Clean-Slate Branch (featureConsensusEntropy)

The new branch started as consensus documentation. That documentation work clarified failure boundaries so sharply that it became a from-scratch implementation effort.

This was not a rename exercise. It was selective reconstruction:

• keep the proven cryptographic core (commit/reveal),
• discard brittle runtime shape,
• port only ideas that survived contact with real network behavior.

In other words: the primitive survived, the convergence model changed.

Hooks-facing RNG APIs such as dice() and random() were among the pieces carried forward and finalized in this architecture.

Breakthrough 1: converge on inputs, not output opinions

The core shift was simple and profound:
do not vote on final entropy values; converge on signed input sets.

Breakthrough 2: proposal-carried leaves + set identities

ExtendedPosition carries:

• myCommitment
• myReveal
• commitSetHash
• entropySetHash

Fast path: normal proposal traffic carries most of what nodes need.
Safety net: SHAMap-backed set identity enables deterministic reconciliation when packets drop or nodes lag.

Breakthrough 3: equality firewall

ExtendedPosition::operator== compares txSetHash only.

That keeps core Tx-set convergence from being held hostage by RNG sub-state timing differences while still allowing entropy sub-state convergence to proceed and reconcile.

Breakthrough 4: sub-states, not a top-level RNG phase

Instead of adding another global phase boundary, the design runs RNG progression inside establish sub-states:

• ConvergingTx
• ConvergingCommit
• ConvergingReveal

This preserved the existing consensus cadence while integrating entropy convergence where it belongs.

Breakthrough 5: SHAMap union convergence

Union merge is monotonic: sets grow as verified leaves arrive.
Scalar opinions can oscillate; verified set growth does not.

And SHAMap mechanics keep reconciliation practical:

• compare roots first,
• walk only diverging branches,
• fetch/merge missing leaves instead of replaying full sets.

So overhead is low on the golden path, with bounded recovery cost when reconciliation is needed.

---

Part IV: Hardening Moves That Made It Viable

The architecture became robust only after concrete hardening steps, each forced by a specific failure mode observed during testnet runs:

1. Proposal proof blobs (sfBlob in SHAMap entries): without these, any peer could inject spoofed commit/reveal entries during a Cold Path fetch. Embedding the proposal signature makes every contribution independently verifiable.
2. Reveal-vs-commit verification (sha512Half(reveal, pubKey, seq) == commitment): without this, a validator could commit to one value and reveal another (grinding attack).
3. UNL enforcement across harvest/build/merge paths: without filtering by UNL membership, non-trusted nodes could contribute entropy and shift the output.
4. Deterministic expected-proposer logic: ties commit-set membership to who actually proposed last round (intersected with UNL), preventing timeouts from waiting for offline validators.
5. Split timeout strategy (3s commit window from round start, 1.5s reveal window from phase entry) with impossible-quorum early exit: without the split, txSet convergence time consumed the reveal budget, leaving no time for reveals to propagate.
6. Deterministic fallback behavior (partial/zero entropy paths): if 80% quorum is met but not all expected proposers arrived, the round proceeds with a partial commitSet rather than discarding available entropy. If quorum is impossible, zero entropy is injected immediately rather than stalling.

Important nuance:

• The design targets full reveal completion for committed contributors (bounded by the 1.5s reveal timeout — not an infinite wait).
• Under timeout/failure conditions, deterministic bounded fallback paths exist so liveness is preserved.

Concrete progression: before the reveal-convergence fixes, a 15-node testnet produced 7 distinct entropy values in the same round (nodes collected different 80% subsets of reveals). After these hardening steps, 20-node testnets reported identical commit-set hashes and ~2.2s convergence with bounded recovery under node loss. (That ~2.2s came from aggressively tuned low-ms settings, including XAHAU_RNG_POLL_MS and tight timeout windows; broader production topologies may need larger windows.)

---

Part V: The Masterstroke (ttCONSENSUS_ENTROPY)

Once nodes converge on the relevant verified input set, final entropy is computed deterministically (sha512Half(sorted_reveals)) and injected as a synthetic pseudo-transaction: ttCONSENSUS_ENTROPY.

This injection happens locally in doAccept(), right before ledger construction. The pseudo-transaction is sorted to execute first in BuildLedger.cpp, so all user transactions and Hooks executing in that same ledger block can consume the entropy via the dice() and random() WebAssembly APIs.

Why no final gossip round on the derived scalar?

Because gossip resolves disagreements.
At this point, the system has already converged on verified inputs; the output function is deterministic. Forcing an extra opinion round adds delay and bandwidth cost without cryptographic benefit.

If a node suffers a local fault and synthesizes the wrong pseudo-tx, its resulting ledger hash will mismatch the network supermajority. Its validations will fail, and it will safely fork off and fetch the correct ledger from peers. Ledger safety is preserved by the validation phase, not the deliberation phase.

---

Safety and Liveness Framing

A useful framing that survived all iterations:

• validation-phase quorum remains the safety anchor for ledger agreement,
• entropy quality/availability is a separate axis that must degrade deterministically under stress.

This matches the formal XRPL LCP framing in Chase & MacBrough (2018): Example 5 captures the key intuition that deliberation outcomes can vary, while fork safety itself is anchored by validation-phase overlap conditions formalized in Theorem 8.

This safety claim is specifically about ledger agreement, not about maximum entropy strength under adversarial withholding.

That distinction prevented a lot of category errors in design discussions.

---

Closing

The final featureConsensusEntropy architecture is "least-bad" in the engineering sense:

• more machinery than naive RNG,
• but each mechanism exists because an observed failure mode forced it.

From ttRNG to dual-tx entropy, to dedicated shuffle phase, to scalar-convergence rejection, the trajectory kept pointing to the same destination:

commit/reveal inputs, SHAMap set identity, union reconciliation, deterministic synthetic injection, and bounded fallback behavior.