Single-bit errors in .blk files triggers partial loss of acknowledged writes

[aphyr]

Observed behavior

In version 2.12.1, with a five-node cluster and replication-factor 5, a Jetstream stream using sync-interval always which experiences single-bit errors in the blk files for a stream, even if those errors are restricted to a minority of nodes, can cause the loss of some or all acknowledged records. For example, this test lost 20,350 acknowledged records from the middle of the test, but retained 299,525 records before and after.

In this run, 159,427 acknowledged records were lost from the the middle. By this I mean that each of those records was acknowledged, never read, and some process observed a record written later by the same process--this violates a claimed NATS ordering guarantee. 287,249 acknowledged records were lost in total; about half were in a lost postfix of the log, rather than in the middle.

A few log entries might be of interest:

[630995] 2025/11/13 22:19:41.162552 [WRN] Filestore [S-R5F-eI3z0amB] Stream state outdated, last block has additional entries, will rebuild
[630995] 2025/11/13 22:19:41.162561 [WRN] Filestore [S-R5F-eI3z0amB] Recovering stream state from index errored: prior state file

[641421] 2025/11/13 22:19:40.939994 [INF]   Starting restore for stream 'jepsen > jepsen-stream'
[641421] 2025/11/13 22:19:41.057012 [INF]   Restored 529,384 messages for stream 'jepsen > jepsen-stream' in 117ms

Expected behavior

NATS has a checksum mechanism which sometimes--but not always!--detects file corruption and refuses to start the node. It should probably prevent silent data loss here too. It's also somewhat worrying that this occurs even though a majority of nodes have totally intact data files!

Server and client version

This is with NATS 2.12.1, and jnats 2.24.0

Host environment

No response

Steps to reproduce

You can reproduce this with commit 3f7d648 of the Jepsen NATS test by running

lein run test-all --nemesis kill,bitflip-file-chunks --time-limit 120 --leave-db-running --version 2.12.1 --sync-interval always --rate 10000 --no-lazyfs --test-count 100

[aphyr]

I've expanded the file corruption tests a bit, and set them up to target different file types. Corruptions in .blk files definitely cause partial data loss, and that loss can be fairly extensive, even if the damage is limited to just 2/5 nodes. Here's another example with the data files, in case it's helpful: https://s3.amazonaws.com/jepsen.io/analyses/nats-2.12.1/20251116T061217-blk-bitflip.zip

This test run also hit a client error I haven't seen before. When publishing a record, the client threw an io.nats.client.JetStreamApiException: "expected sequence does not match store [10077]".

[aphyr]

The tests above lots tons of data, but it looks like .blk corruptions can also cause more subtle loss. In this run, we lost just a single record (96-4) out of 1359025 acknowledged writes. This miiiiight be a different bug? https://s3.amazonaws.com/jepsen.io/analyses/nats-2.12.1/20251116T030143-blk-bitflip-single-loss.zip

[neilalexander]

Thanks for the report, we will look into this further.

Just for awareness, detecting errors in the filestore can happen in more than one way: typically it's by a checksum that's stored per-message, then there are additional checksums on the state files. If on recovery a message checksum is found to be faulty, I believe the current behaviour is to truncate the remainder of that block back to the faulty sequence. The earlier the message is in a given block, or if the following messages are smaller, then we may be throwing away a larger postfix of messages from that block as a result.

If a message checksum fails, we can't know whether that is because of a checksum failure in the message content itself, or whether it's in the metadata describing for example the message length. If it is the latter then skipping over the corrupted message to the correct next index becomes considerably more difficult, at which point we would be unable to recover the rest of the block anyway.

Since following blocks could be fine, this would indeed create a gap of lost data in the middle of the stream (and should report in the stream info's num deleted count). We don't currently have an active method of detecting this mid-stream and reconciling from peers, although I think there are certain specific cases (i.e. where we truncate the postfix of the last message block in particular) where catchup from a peer would do the right thing by default, same as if a node was offline for a while and was naturally behind the leader.

Corruption in the state files is usually easier to deal with as we can usually throw those away and rebuild them by rescanning the stream, if no other corruption occurs.

---

As an aside, at present it is not common for JetStream to refuse to start in these scenarios. There could be any number of other perfectly functional (non-corrupted) streams on a node that would also be affected by refusing to start, and the only actionable outcome for an operator would be to delete the corrupted stream store from the disk of that node to get the node up and running again. In a replicated stream that may be OK with a backfill/catchup from another node if other cluster nodes are otherwise running normally, but with a non-replicated stream then that is less than ideal.

It is also worth mentioning that it is possible to "mirror" one stream into another, even when the destination mirror is in a different cluster or JetStream domain, in cases where the system has been extended with e.g. leafnodes. Mirrors do not replicate deletes and can be configured with their own retention policies and often help to feed into a disaster-recovery or data backup strategies, but understand that this does not mitigate a live-loss scenario as you outline.

[aphyr]

Errors in the .blk files on a minority of nodes can also cause split-brain: acknowledged records can go missing on a majority of nodes, but be present on some. Missing records can be those written before, in the middle, or after values which were successfully read. For example, in this run readers talking to node n2 observed every acknowledged record...

But readers talking to n5 thought 1,167,167 out of 1,479,661 acknowledged records were lost!