[Feature] Support program upgradability. #2655

d0cd · 2025-03-19T01:07:12Z

[NO NEW COMMITS ARE ALLOWED AS OF 5/27/25]

DO NOT MERGE UNTIL BASE IS UPDATED TO staging

Motivation

This PR

supports program upgradability, by allowing programs with constructors to be updated. Constructors are immutable, allowing developers to implement custom update logic for their applications.
modifies the program checksum to use SHA256 and changes its type from field to a [u8; 32u32]
strictly defines program equality to also include order
adds an optional program_checksum field to Deployment, which implicitly versions the object.
updates the deployment ID calculation to use the program checksum. Benchmarks for the checksum calculation are included below.

Test Plan

Process-level tests were added to verify the syntactic restrictions on upgrades.
VM/Ledger-level tests were added to verify upgrade semantics.

Benchmarks

Run on a 2021 MBP w/ an M1 Max & 32GB RAM

SHA3_256 Hash - input size 1000 bytes - [3.4339 µs 3.4435 µs 3.4539 µs]
SHA3_256 Hash - input size 10000 bytes - [28.346 µs 28.500 µs 28.747 µs]
SHA3_256 Hash - input size 100000 bytes - [280.97 µs 282.03 µs 283.11 µs] (max program size)
SHA3_256 Hash - input size 1000000 bytes - [2.7798 ms 2.7846 ms 2.7907 ms]

ledger/store/src/transaction/deployment.rs

synthesizer/process/src/stack/deploy.rs

ledger/store/src/helpers/rocksdb/internal/id.rs

raychu86 · 2025-03-20T18:23:28Z

ledger/store/src/transaction/deployment.rs

@@ -186,7 +197,9 @@ pub trait DeploymentStorage<N: Network>: Clone + Send + Sync {
        atomic_batch_scope!(self, {
            // Store the program ID.
            self.id_map().insert(*transaction_id, program_id)?;
-            // Store the edition.
+            // Store the edition for the transaction ID.
+            self.id_edition_map().insert(*transaction_id, edition)?;


This will insert a 0 edition for new programs, but have a missing item for existing programs

raychu86 · 2025-03-20T18:25:17Z

ledger/store/src/helpers/memory/transaction.rs

@@ -92,6 +92,8 @@ impl<N: Network> TransactionStorage<N> for TransactionMemory<N> {
 pub struct DeploymentMemory<N: Network> {
    /// The ID map.
    id_map: MemoryMap<N::TransactionID, ProgramID<N>>,
+    /// The ID edition map.
+    id_edition_map: MemoryMap<N::TransactionID, u16>,
    /// The edition map.
    edition_map: MemoryMap<ProgramID<N>, u16>,


We may want to rename this to latest_edition_map.

Should be fine on the rocksdb level, but will need to do some testing and ensure existing ledgers boot and operate properly.

ledger/store/src/transaction/deployment.rs

synthesizer/process/src/stack/helpers/check_update.rs

synthesizer/src/vm/verify.rs

…ability

ledger/block/src/transaction/merkle.rs

…leHQ/snarkVM into feat/improve-tree-logic

ledger/block/src/transaction/deployment/mod.rs

ledger/block/src/transaction/deployment/bytes.rs

ledger/block/src/transaction/deployment/mod.rs

ledger/block/src/transaction/merkle.rs

ledger/store/src/transaction/mod.rs

synthesizer/src/vm/finalize.rs

synthesizer/src/vm/mod.rs

synthesizer/src/vm/verify.rs

synthesizer/src/vm/deploy.rs

vicsn · 2025-04-08T10:26:00Z

synthesizer/src/vm/finalize.rs

@@ -786,6 +790,8 @@ impl<N: Network, C: ConsensusStorage<N>> VM<N, C> {
    /// - The transaction is producing a duplicate output
    /// - The transaction is producing a duplicate transition public key
    /// - The transaction is another deployment in the block from the same public fee payer.
+    /// - The transaction is an execution for a program following its deployment or redeployment in this block.


I'm still in favour of aborting executions which reference a state root which is older than the latest program upgrade corresponding to any of the transitions in the execution.

I think this is a fair point.
The lookup could be done by:

Look up the latest transaction ID that deployed or upgraded a program.

Look up the corresponding block height for the program.

Look up the corresponding block height for the state root.

Verify that the state root's block height is less than the program's last deployed/updated height.

One thing to consider is that this check would create a dependence on the state root for "non-record" transactions. Concretely, exchanges and other developer services tend to "fix" a state root to reduce the number of network calls needed for their transactions. It could change the UX for consumer software but is not an unreasonable thing to do.

I'll leave it to reviewers to arbitrate.

Sounds great. I can't imagine this to be a major performance bottleneck.

However, we do need to check the # program for each transition. So maybe a cache is warranted.

CC @raychu86

One more reason why this is a good idea:

If a user's local state is behind the network, then they might use an outdated edition or checksum value as witness, and as a result, the proof will fail to verify with an extremely opaque error message.

Now, unfortunately, users are used to transactions failing for opaque reasons. But we can imagine a point where explorers surface this information to users, or developers see this in validator logs for debugging purposes.

It would be a lot better if the user or developer would see e.g. "Execution failed due to the use of a state root older than the latest program upgrade."

For the same reason as my last message above (verifier witness data may change after a program update), we should clear executions from the partially_verified_transactions cache when the underlying program for any of its transitions has been updated.

This should happen already before a block is finalized, so that if a program is updated, the next execution within the same block might now be rejected/aborted during finalization.

Maybe this ends up being superfluous if we add a separate explicit check for the state root, but it doesn't hurt to evict a cache early.

Perhaps we can pull in ecosystem developers into this code review weigh in?

I like it! I'll share a plan with you.

Note that in any case we will need to clear appropriate executions from partially_verified_transactions - though the location where it is cleared might depend on that user feedback.

I made a branch with the check (diff)
I'll make a note of it in the tracking issue as well.

Refreshed the above branch to do the check in should_abort_transaction

I genuinely would love to require a recent state root to ensure freshness.

However, I do feel for offline wallets, cold storage schemes, and HSMs (in the future). Namely, that they do not have the privilege or luxury to go and fetch live state like this on a continual basis. And a custodian servicing millions of addresses for their customers does not have the bandwidth to stay on top of live refreshes like this (with today's state of Aleo infrastructure and tooling).

As such, I strongly recommend keeping the design in its current form and re-evaluating down the line.

I do agree that fetching live state is a pain for offline wallets, HSMs, etc.
However, these solutions will already need to do this to spend records.

The proposed design can use any state root after the the program has been upgraded.
Moreover, it's unlikely for programs to get upgraded frequently, even once a month would be a lot.
And so is unlikely to add meaningful overhead to existing requirement for state root fetching.

This restriction does improve security for users, limiting the ways a malicious developer could "rug" a user.

Removes `PartialEq` and `Eq` on Map `V` traits

raychu86 · 2025-04-10T22:12:40Z

synthesizer/program/src/logic/command/get_or_use.rs

-    /// The default value.
-    default: Operand<N>,
+    /// The operands.
+    operands: [Operand<N>; 2],


Why the change here? It felt more clear with the direct variables key and default.

At the very least, the documentation of operands should reflect the structure.

Ditto with the other commands that have this unification

This was needed to support the operands() on commands without allocating Vecs
The instructions are actually implemented in a similar way.
TBH the implementations of both instructions and commands needs some refactoring to standardize.

For additional context, we need to iterate through all of the operands in a program and check that they don't match Operand::Checksum or Operand::Edition before a specific consensus version. It would be ideal if we can do this without cloning and allocating.

Signed-off-by: vicsn <[email protected]>

Improves safety & readability on transaction root derivation logic

…ability

d0cd mentioned this pull request Mar 19, 2025

[Track] Program Upgradability #2654

Open

21 tasks

d0cd requested review from vicsn and raychu86 and removed request for vicsn March 19, 2025 15:49

d0cd force-pushed the feat/program-updatability branch from cc239a5 to 9d94734 Compare March 19, 2025 22:33

d0cd commented Mar 20, 2025

View reviewed changes

ledger/store/src/transaction/deployment.rs Outdated Show resolved Hide resolved

d0cd commented Mar 20, 2025

View reviewed changes

synthesizer/process/src/stack/deploy.rs Show resolved Hide resolved

d0cd added 9 commits March 20, 2025 11:15

Support updatability

6d284b4

Update deployment store

ac8545c

Fix deployment (de)serialization

30db805

Add benches and add migration TODOs

434a3f8

Cleanup

90f2621

Cleanup up Deployment

86810f5

Improve deployment store API

1a76586

Update transaction store API

75d047a

Enable program checksums in the deployment

3b02904

d0cd force-pushed the feat/program-updatability branch from f27bcae to 3b02904 Compare March 20, 2025 18:17

raychu86 reviewed Mar 20, 2025

View reviewed changes

ledger/store/src/helpers/rocksdb/internal/id.rs Outdated Show resolved Hide resolved

raychu86 reviewed Mar 20, 2025

View reviewed changes

ledger/store/src/transaction/deployment.rs Outdated Show resolved Hide resolved

raychu86 reviewed Mar 20, 2025

View reviewed changes

synthesizer/process/src/stack/helpers/check_update.rs Outdated Show resolved Hide resolved

raychu86 reviewed Mar 20, 2025

View reviewed changes

synthesizer/src/vm/verify.rs Outdated Show resolved Hide resolved

d0cd added 6 commits March 20, 2025 12:42

Improve comments, add checksum map

5112657

Implement migrations

de65776

Fix test

69f2583

Cleanup comments

7b36570

Merge branch 'feat/constructors-and-operands' into feat/program-updat…

2eb82ba

…ability

Feedback

462920d

howardwu reviewed Mar 28, 2025

View reviewed changes

ledger/block/src/transaction/merkle.rs Outdated Show resolved Hide resolved

d0cd and others added 8 commits March 28, 2025 14:42

Feedback

04d13cd

Merge branch 'feat/program-updatability' of https://github.com/Provab…

266fbff

…leHQ/snarkVM into feat/improve-tree-logic

Remove PartialEq and Eq on Map traits

0236435

Feedback

b624f38

Feedback, redefine program checksums

018076a

Feedback

08cdc22

Fix tests

207f891

Restrict to one upgrade/deployment per block

cacd25a

vicsn requested changes Apr 8, 2025

View reviewed changes

vicsn and others added 6 commits April 8, 2025 13:33

Merge pull request #2669 from ProvableHQ/remove/eq-on-maps

2eea916

Removes `PartialEq` and `Eq` on Map `V` traits

Feedback

b97c783

Fix

005cd60

Add to find/get API

4fadf23

Refactor based on feedback

0e274f1

Further restrict new syntax before migration

c618b26

raychu86 reviewed Apr 10, 2025

View reviewed changes

Fix

be0a27c

d0cd requested a review from vicsn April 15, 2025 16:46

Add zero check for edition

a225903

d0cd requested a review from raychu86 April 18, 2025 15:32

howardwu and others added 9 commits April 18, 2025 15:37

Add additional fee_index check

05f30a5

Resolve merge conflict

493f29f

Adds extra deployment size checks

bfa667f

Update ledger/block/src/transaction/merkle.rs

13afd8e

Signed-off-by: vicsn <[email protected]>

Feedback

d33c5ef

Merge pull request #2668 from ProvableHQ/feat/improve-tree-logic

bcaf360

Improves safety & readability on transaction root derivation logic

Merge branch 'feat/constructors-and-operands' into feat/program-updat…

57b6790

…ability

Bump to V8

24b8453

Fix tests

0fc9566

[Feature] Support program upgradability. #2655

Are you sure you want to change the base?

[Feature] Support program upgradability. #2655

Uh oh!

Conversation

d0cd commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Test Plan

Benchmarks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vicsn Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d0cd Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vicsn Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vicsn Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d0cd Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

d0cd commented Mar 19, 2025 •

edited

Loading

vicsn Apr 8, 2025 •

edited

Loading

d0cd Apr 8, 2025 •

edited

Loading

vicsn Apr 8, 2025 •

edited

Loading

vicsn Apr 8, 2025 •

edited

Loading

d0cd Apr 10, 2025 •

edited

Loading