Database corrupted when restoring from backup

[fuchstim]

Bug Description

Hey all, I'm currently working on integrating Litestream into one of our production workloads but I keep running into seemingly random database corruption issues.

I set up Litestream following this guide. My service is deployed as a statefulset; an init container restores the most recent replica from S3/GCS/ABS on startup then a sidecar container continuously replicates new changes to blob storage.

However, every now and again I'll start seeing database disk image is malformed errors in my application. Running an integrity check always returns warnings along the lines of

sqlite> pragma integrity_check;
pragma integrity_check;
integrity_check
---------------------------------------------------------------------
wrong # of entries in index my_table_uid_idx
wrong # of entries in index sqlite_autoindex_my_table_1

There don't appear to be any error logs coming from litestream, neither during the replication nor the restoration. There also haven't been any pod crashes or unclean shutdowns that could explain the corruptions.

I understand that this ticket is a bit thin on information, I'll try to enable trace logging and update the issue with more information if the issue occurs again. Unfortunately I haven't been able to manually trigger the corruption, so it might be a while until I have any more details. I thought I'd raise the issue anyways in case anyone has seen this before and there's a known fix.

Environment

Litestream version:

0.5.8

Operating system & version:
litestream/litestream:0.5.8 Docker image

Installation method:
Kubernetes init container to restore / sidecar container to replicate, following this setup

Storage backend:
S3, GCS, ABS

Steps to Reproduce

I haven't been able to reproduce this, it seems to happen randomly.

Configuration

litestream.yml

addr: ":9090"
dbs:
  - path: /mnt/db/database.db
    replica:
      url: s3://my-bucket/my-service/database.db

Logs

No logs outside of the usual compaction complete and l0 retention enforced. Will enable trace logs and try to capture more information the next time this occurs.

Additional Context

Running sqlite version 3.51.1 2025-11-28 17:28:25 281fc0e9afc38674b9b0991943b9e9d1e64c6cbdb133d35f6f5c87ff6af3alt1 (64-bit). The main application is a Go app using the mattn/go-sqlite3 driver. The database is stored in an emptyDir mount i.e. there is no persistent state across pods.

[fuchstim]

Not sure if this is a relevant piece of information, but the database I'm replicating was 'adopted' by litestream (i.e. we enabled litestream after the fact rather than at the time the DB was first created)

[fuchstim]

Looks like the issue occurred again overnight, this time I was able to gather some more details:

• The corruption did not coincide with a pod restart
• Running pragma integrity_check; on the source DB in the (still alive) pod returns ok
• I was able to pinpoint the moment of corruption using the point-in-time restore feature:

$ litestream restore -timestamp "2026-02-24T17:07:00Z" -o $(pwd)/17.07.db s3://bucket/database.db | tee logs-17.07.txt
$ sqlite3 17.07.db "pragma integrity_check;"
ok

$ litestream restore -timestamp "2026-02-24T17:08:00Z" -o $(pwd)/17.08.db s3://bucket/database.db | tee logs-17.08.txt
$ sqlite3 17.08.db "pragma integrity_check;"
*** in database main ***
Tree 8 page 82463 cell 17: Rowid 2902660 out of order
row 199107 missing from index my_table_deleted_at_idx
row 199107 missing from index my_table_resource_version_idx
wrong # of entries in index my_table_org_resource_id_idx
wrong # of entries in index my_table_deleted_at_idx
wrong # of entries in index my_table_resource_version_idx
wrong # of entries in index my_table_uid_idx
wrong # of entries in index sqlite_autoindex_my_table_1

I've uploaded TRACE level logs here. This includes logs from the replication sidecar between 17:00 and 17:10, as well as from the restore process for the PIT restores.

[fuchstim]

Another observation; it seems that in the corrupted DB (the one restored at 17:08) there's a duplicate row (notice the identical ROWID, also there's a unique constraint on the _uid column):

sqlite> select ROWID, _uid, _updated_at from my_table where {...};
ROWID|_uid|_updated_at
2902658|2f293c42-c8cc-4b49-b87d-86f312a6034a|2026-02-24 17:07:17.92+00:00
2902658|2f293c42-c8cc-4b49-b87d-86f312a6034a|2026-02-24 17:07:17.92+00:00

[fuchstim]

Just caught another instance:

$ litestream restore -timestamp "2026-02-25T11:23:00Z" -o $(pwd)/11.23.db s3://bucket/database.db
$ sqlite3 11.23.db "pragma integrity_check;"
ok

$ litestream restore -timestamp "2026-02-25T11:24:00Z" -o $(pwd)/11.24.db s3://bucket/database.db
$ sqlite3 11.24.db "pragma integrity_check;"
row 188104 missing from index my_table__deleted_at_idx
row 188104 missing from index my_table__resource_version_idx
wrong # of entries in index my_table__deleted_at_idx
wrong # of entries in index my_table__resource_version_idx

Here are the litestream replicate logs during that time: https://gist.github.com/fuchstim/01dcfee6b84de564c845f5446fa0b0df

[fuchstim]

Curiously it seems the backup recovered at some point:

$ litestream restore -timestamp "2026-02-25T15:13:00Z" -o $(pwd)/15.13.db s3://bucket/database.db
$ sqlite3 15.13.db "pragma integrity_check;"
wrong # of entries in index my_table__uid_idx
wrong # of entries in index sqlite_autoindex_my_table_1

$ litestream restore -timestamp "2026-02-25T15:14:00Z" -o $(pwd)/15.14.db s3://bucket/database.db
$ sqlite3 15.14.db "pragma integrity_check;"
ok

replicate logs: https://gist.github.com/fuchstim/bec984e17f951a2be1a5fa5565b4739b

[fuchstim]

In an attempt to reproduce the error locally I set up this repo: https://github.com/fuchstim/litestream-corruption-reproduction
This closely mirrors the setup including the DB schema, data access patterns, and deployment model (although this one uses docker-compose instead of kubernetes).

After running it for a while I was able to reproduce the error once, although I can't seem to reliably repeat it e.g. by frequently stopping / restarting the stack.

app-1                 | 2026/02/26 15:10:17 integrity error: wrong # of entries in index data_name_idx
app-1                 | 2026/02/26 15:10:17 integrity error: row 97113 missing from index data_name_idx
app-1                 | 2026/02/26 15:10:17 integrity check failed: database integrity check failed with 2 error(s)
app-1 exited with code 1

EDIT:
I've also added a dump of the S3 files from the above test run to the repo (needs git LFS to be installed): https://github.com/fuchstim/litestream-corruption-reproduction/blob/master/s3-dump.tar.gz

[corylanou]

Thanks for the detailed trace logs, reproduction repo, and all the debugging work — it was incredibly helpful in tracking this down.

We believe we've identified the root cause and have a fix in PR #1166. The TL;DR:

The bug: PageMap() in wal_reader.go was returning a map of page numbers to WAL file offsets. Later, writeLTXFromWAL() and writeLTXFromDB() would re-read the actual page data from those offsets. Between those two reads, SQLite's PASSIVE checkpoint could rewrite or truncate the WAL — so the re-read would capture checkpoint-modified (garbled) page data into the LTX files uploaded to replicas.

The fix: PageMap() now buffers the page data (map[uint32][]byte) during the initial checksum-verified read instead of recording offsets for later re-read. This eliminates the race window entirely.

Your trace logs confirmed this. In gist 1, during the txid 0x2de sync there's a 55ms gap between encoding page 73153 (at 17:07:18.316) and page 82934 (at 17:07:18.371), with the PASSIVE checkpoint firing at 17:07:18.391. That's the race window — the checkpoint modifies WAL content at the offsets that were already recorded but not yet re-read. All three of your trace logs show this same sync → checkpoint → WAL restart sequence.

This also explains:

• The "missing from index" errors — a data page gets captured at version N but the corresponding index page at version N-1 (or vice versa) because the checkpoint overwrote one but not the other before re-read
• The self-healing you observed (corrupt at 15:13, clean at 15:14) — once the correct version of a corrupted page is written in a later transaction, restoring past that point overwrites the bad page
• The intermittent nature — requires the checkpoint to fire at exactly the right moment during sync

We also ran your reproduction setup against the fix and it appears to resolve the issue.

Would you mind taking a look at PR #1166 and seeing if the analysis lines up with what you've been observing? Given how much context you have on this from the debugging side, your review would be really valuable.