Skip to content

Conversation

@devanbenz
Copy link

@devanbenz devanbenz commented Dec 22, 2025

We are experiencing a bug where we have null shards for a given ShardGroup in the boltdb metadata. This seems to show up during backup -> restore. Currently we have not been able to reproduce organically. This PR adds logging to backup and restore handlers for when a given ShardGroup metadata is like the following example:

            {
              "ID": 789,
              "StartTime": "2025-12-01T00:00:00Z",
              "EndTime": "2025-12-08T00:00:00Z",
              "DeletedAt": "0001-01-01T00:00:00Z",
              "Shards": null,
              "TruncatedAt": "0001-01-01T00:00:00Z"
            },

Notice there is no DeletedAt and the Shards are null. This is what we want to avoid.

Example for backup:

ts=2025-12-22T23:09:24.489579Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=2 start-time=2025-12-22T17:23:00.000000Z end-time=2025-12-22T17:24:00.000000Z
ts=2025-12-22T23:09:24.489655Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=3 start-time=2025-12-22T18:11:00.000000Z end-time=2025-12-22T18:12:00.000000Z
ts=2025-12-22T23:09:24.489671Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=4 start-time=2025-12-22T18:12:00.000000Z end-time=2025-12-22T18:13:00.000000Z
ts=2025-12-22T23:09:24.489678Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=5 start-time=2025-12-22T22:58:00.000000Z end-time=2025-12-22T22:59:00.000000Z

Example for restoration:

ts=2025-12-22T23:20:24.971567Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=2 start-time=2025-12-22T17:23:00.000000Z end-time=2025-12-22T17:24:00.000000Z
ts=2025-12-22T23:20:24.971590Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=3 start-time=2025-12-22T18:11:00.000000Z end-time=2025-12-22T18:12:00.000000Z
ts=2025-12-22T23:20:24.971594Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=4 start-time=2025-12-22T18:12:00.000000Z end-time=2025-12-22T18:13:00.000000Z
ts=2025-12-22T23:20:24.971602Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=5 start-time=2025-12-22T22:58:00.000000Z end-time=2025-12-22T22:59:00.000000Z

@devanbenz devanbenz marked this pull request as ready for review December 22, 2025 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants