go/oasis-node/cmd/storage: Add create and import checkpoint cmd

[martintomazic]

Closes #6423

• Write our own version of BootstrapState
  • The node creating checkpoint can dump untrusted metadata, that will be used for initializing cometbft stores after importing the checkpoint.
  • We could also make it trustless, if we want to get rid of the snapshots entirely and only store checkpoints + storage diffs and blocks. :)
• Fix TODOs depending on the review feedback.

[netlify]

✅ Deploy Preview for oasisprotocol-oasis-core canceled.

Name	Link
🔨 Latest commit	f082fcd
🔍 Latest deploy log	https://app.netlify.com/projects/oasisprotocol-oasis-core/deploys/69a95948f21a130008af6a91

[martintomazic]

Works! :)

The only thing that is impractical is finding corresponding runtime rounds to given consensus height and the fact that bootstrap "eats" one height as described.

Finally, one should be very careful with creation/import height/rounds so that you have all relevant light history for the runtime checkpoints you are importing.

[martintomazic]

Maybe use default undefined round (aka max uint64), alternative is cmd.Flags().Changed("height").

Update: Alternative is explicit --consensus flag or possible consensus/runtime sub-commands. No height/round could also mean latest height. -all flag with --height would be also interesting if it would find corresponding runtime rounds for the given height.

[peternose]

Currently things are not fine, as you can do consensus and runtime checkpoints at the same time. Furthermore, this might be confusing for users, e.g.., do they need to set height, round, both?

[martintomazic]

Yes I left it intentional for now. I can easily allow only one at the time. The question is would using sub-commands make things even clearer? Also any preference for what should omitting height/round do?

Observe also that if you want to be able to create checkpoints for multiple heights/versions (same NodeDB) the import command also grows in complexity.

Finally, one should be very careful for which height to specific runtime round combinations you are creating as you can easily get left missing missing runtime light blocks, and therefore stuck runtime state restore.

We could also make import command use consensus/runtime/height/round and import the actual directories created one by one to make it symmetric to create.

Any pref?

[martintomazic]

Creating checkpoints from the penultimate snapshot, is dominated by the Sapphire checkpoint creation.

With 6 chunker threads current projection is 5-7 hours (will update). Import is a matter of minutes.

[martintomazic]

Added unit and e2e tests, fixed empty state corner case and improved code quality.

Two minor things left to discuss:

• Exact API of this command / default values actions: see go/oasis-node/cmd/storage: Add create and import checkpoint cmd #6454 (comment)
• Consider refactoring checkpoint abstraction as proposed: Improve storage.mkvs.checkpoint abstractions #6467 (follow-up).

[codecov]

Codecov Report

❌ Patch coverage is 59.29204% with 138 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.56%. Comparing base (c9a4b8e) to head (06b5cc4).

Files with missing lines	Patch %	Lines
go/oasis-node/cmd/storage/checkpoint.go	58.43%	73 Missing and 65 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6454      +/-   ##
==========================================
- Coverage   64.73%   64.56%   -0.18%     
==========================================
  Files         699      700       +1     
  Lines       68246    68581     +335     
==========================================
+ Hits        44179    44279     +100     
- Misses      19060    19183     +123     
- Partials     5007     5119     +112     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[peternose]

When I import a consensus checkpoint, I get few lines of the following error. Afterwards, blocks execute normally.

{"caller":"grpc.go:194","err":"failed to get consensus status: failed to fetch current block: cometbft: block query failed: height 28800866 must be less than or equal to the current blockchain height 0","level":"error","method":"/oasis-core.NodeController/GetStatus","module":"grpc/internal","msg":"request failed","req_seq":15,"ts":"2026-02-24T13:00:28.934344662Z"}

[peternose]

Maybe creating a struct checkpointer would be better, as you could create multiple checkpoints with the same struct, e.g.

cp.Create(ctx, 1, "/checkpoints/1")
cp.Create(ctx, 2, "/checkpoints/2")
...

or even without outputDir if accepted in the constructor. The new struct might also be easier to test and could be decoupled from the commands.

[martintomazic]

update:

The new struct might also be easier to test and could be decoupled from the commands.

newCheckpointer + cp.create (proposed) = createCheckpoints (current) so this is only a matter of style.

As usual I prefer an explicit function over abstractions until multiple methods share same parameter set.

---

The question is do we want to allow multiple checkpoint heights/rounds for the same NodeDB.

If you want to make this generic helper I think this would fit inside checkpoint package.
See ( #6467):

// Consider using functional optional arguments to shorten args list.
CreateCheckpoint(ctx context.Context, ndb db.NodeDB, store Store, root node.Root, chunkSize uint64, chunkerThreads uint16) (*Metadata, error)

// Maybe add CreateAllCheckpoints as well and avoid passing root there.

[peternose]

Yes, it is a matter of style by the quality/complexity of the code. And more parameters you have, the harder is to follow. And as suggested, compare code if you need to create multiple checkpoints for different versions with a function or with a method call.

[peternose]

Currently things are not fine, as you can do consensus and runtime checkpoints at the same time. Furthermore, this might be confusing for users, e.g.., do they need to set height, round, both?

[martintomazic]

When I import a consensus checkpoint, I get few lines of the following error. Afterwards, blocks execute normally.

Nice catch. Yes this is also how CometBFT checkpoint import works, but found a fixup regardless :).

The more annoying thing that I find is that you technically cannot import a checkpoint for the latest height, so probably adding extra validation + documenting this in the command would be beneficial, instead of unexpected error.

[martintomazic]

Ready for a second review.

As you spotted I am "abusing" checkpoint.FileStore abstraction (#6467) bound to one concrete NodeDB, that should be able to create and import checkpoints for different versions.

However, this command is technically not create/import checkpoint but rather create/import state/snapshot. Indeed 99% is creating and importing checkpoints and using checkpoint.FileStore for encoding the snapshot structure. Nevertheless, CometBFT bootstrap metadata has nothing to do with the checkpoints alone.

For this reason I have created my helpers (stateless), so that they can also be easily refactored, possibly moved to checkpoint package one day. Until this command is the only client I am not sure it makes sense. Improving checkpoint package abstractions probably does. :)

Let's align on the user facing API and sanity checking the inputs:

1. go/oasis-node/cmd/storage: Add create and import checkpoint cmd #6454 (comment)
   • The simpler the better given we will the only user of this command:
2. Should we find a better name for this command given that is not just creating/importing checkpoints? Not sure.

[peternose]

Ready for a second review.

Will have a look. Merge after we release 26.0.