Creating an incremental snapshot from a full snapshot produces an invalid snapshot

[willhickey]

Operators creating a snapshot for the 2025-12-06 testnet restart reported this error during validator startup:

[2025-12-10T18:53:04.177476879Z ERROR solana_runtime::bank] Verifying accounts failed: accounts lattice hashes do not match, expected: iHkBUhPHs3CspqNsxirq3oYaKeTPawB4E6jUH2nn1wr, calculated: 5ANdAkiQwFCQGkG5PNZeXz5LVpPbbtgw37nEL5xUw6sM

It is caused by using only a full snapshot and the ledger to generate the restart snapshot. Using a full snapshot, an incremental snapshot, and the ledger to create a new incremental snapshot produces a working restart snapshot.

Everything necessary to reproduce the error is in tse1:~/lattice-hash-mismatch.tar.gz

[willhickey]

I ran through this a bit on my dev machine. The difference is from account Ccv97Ew3uBtjmRd5kPiA26o1nU3VpbriWySCCEKArmH. IIRC the problem could be reproduced, but then eventually wouldn't reproduce anymore, which makes me think the non-destructive access to rocksdb was actually modifying it (compacting? cleaning?)

agave-ledger-tool -l ledger accounts \
    --snapshots source-snapshots/full/ \
    --incremental-snapshot-archive-path new-snapshots/from-incremental/ \
    --account Ccv97Ew3uBtjmRd5kPiA26o1nU3VpbriWySCCEKArmH \
    --output json

{
  "accounts": [],
  "summary": {
    "num_accounts": 0,
    "data_len": 0,
    "num_executable_accounts": 0,
    "executable_data_len": 0
  }
}


agave-ledger-tool -l ledger accounts \
    --snapshots source-snapshots/full/ \
    --incremental-snapshot-archive-path new-snapshots/from-full/ \
    --account Ccv97Ew3uBtjmRd5kPiA26o1nU3VpbriWySCCEKArmH \
    --output json

{
  "accounts": [
    {
      "pubkey": "Ccv97Ew3uBtjmRd5kPiA26o1nU3VpbriWySCCEKArmH",
      "account": {
        "lamports": 1985636480,
        "data": [
          "",
          "base64"
        ],
        "owner": "11111111111111111111111111111111",
        "executable": false,
        "rentEpoch": 18446744073709551615,
        "space": 0
      }
    }
  ],
  "summary": {
    "num_accounts": 1,
    "data_len": 0,
    "num_executable_accounts": 0,
    "executable_data_len": 0
  }
}

[kskalski]

eventually wouldn't reproduce anymore, which makes me think the non-destructive access to rocksdb was actually modifying it (compacting? cleaning?)

With ledger-tool from 3.1 there is some possibility for that (it uses "secondary access"), with one from master much less so (it uses "read only access"). We observed that the former does something with the files upon opening (their size changes), but I think neither would do compaction.