Subtree proofs, landmarks, and signatureless certificates by lukevalenta · Pull Request #96 · cloudflare/azul

lukevalenta · 2025-08-26T00:45:16Z

This PR adds support for subtree proofs, landmarks, and signatureless certificates. I recommend going commit by commit as some of them do refactoring.

Closes #64

Some features that I haven't added yet, but will file issues for in case we want to add them in the future:

Make spki_der optional in /get-certificate requests, and fetch the unhashed SPKI from the bootstrap entry to construct the signatureless cert. This would be useful for testing, but probably not necessary for production.
Add a 'landmark bundle' endpoint to bundle up landmark subtree inclusion proofs to some signed checkpoint. If we do this we could also do the batched consistency proofs (@rozbb has also thought about this, at least for consistency proofs: WIP: Batched Merkle Audit Paths C2SP/C2SP#18).

bwesterb

See previous comments on PR.

bwesterb

Feel free to merge already if that makes future PRs smaller.

Later, we'll use the same subroutine to generate both types of proofs, so it's useful to merge them into a single type.

This commit does not add any new functionality, but does some refactoring like adjusting functions and APIs. * Rename 'check_{inclusion,consistency}' functions to 'verify_{inclusion,consistency}' for closer alignment to the spec. * Rename function parameters for consistency (e.g., use 'n' for larger tree, 'm' for smaller tree, 'leaf_index' for leaf index) * Add '_recursion' suffix to '{inclusion,consistency}_proof[_indexes]' functions and make them private (these will be replaced in a later commit). Rename 'prove_{inclusion,consistency}' to '{inclusion,consistency}_proof' for closer alignment with naming in spec, and add corresponding '*_indexes' functions that do bounds checks for safe public consumption. * Add ProofError type to help distinguish between proof errors and server errors. * Move TlogError enum to the top of the source file.

* Add Subtree struct to represent a valid subtree. (More functionality will be added later.) * Add support for subtree inclusion and consistency proof verification. * Replace recursive proof verification functions with ones aligned with the specifications.

* Add _internal suffix to subtree_hash and subtree_indexes functions. These will get removed later. * Add subtree_hash_indexes, subtree_hash, and tree_hash_indexes functions with better API for public consumption.

SUBTREE_SUBPROOF generalizes the logic to compute inclusion and consistency proofs, so we can use the `Subtree::subproof` function to compute all of the various proof types. * Add support for subtree consistency proofs. * Deduplicate subtree recursion logic with 'walk_subtree' helper. * Deduplicate merkle tree hash logic with 'walk_hash' helper. * Remove all of the _internal/_recursion helper functions as those have been replaced.

* Add subtree inclusion and consistency proof support in log_ops. * Update prove_inclusion and prove_consistency to verify the computed proofs, fetching additional tiles from object storage as needed. The subtree inclusion proofs will be used for computing Merkle Tree Certificate 'mtcProof' signatures, and the subtree consistency proofs will be useful for constructing proofs that landmark subtrees are consistent with some checkpoint.

* Add support for landmark checkpoints. * Update /prove-inclusion to compute subtree inclusion proofs within landmark subtrees. * Add checkpoint callback function that gets called every time sequencing completes with the new tree size and old and new timestamps. Importantly, if the callback fails, it does NOT count as a fatal sequencing error. This is used for updating the landmark sequence for the MTC sequencer. * Rename validity_interval_seconds to max_certificate_lifetime_secs.

- Convert MTC /prove-inclusion endpoint to /get-certificate to return a signatureless certificate. - Add read_leaf function to read and verify the needed tiles and then extract the data at a single leaf index. This is used to fetch a MerkleTreeCertificateLogEntry in order to construct a signatureless certificate. - Add support for serializing a signatureless certificate and constructing the MTCProof from a subtree inclusion proof.

Use language from ietf-plants-wg/merkle-tree-certs#139 which aims to clarify some ambiguities.

rozbb

Pretty minor notes. Nice!

We have to fetch additional tiles (to compute the leaf and landmark subtree hashes) to validate subtree inclusion proofs right after we've constructed the proof. Remove validation from the hot path as we have bettter testing now and don't expect to create invalid proofs.

cjpatton

Working my way through, just a few minor things so far.

* Use debug_assert instead of assert in tlog library * Improve documentation and variable naming for stored hash index * Add link to spec for subtree definition

lukevalenta requested review from bwesterb and cjpatton August 26, 2025 00:45

lukevalenta self-assigned this Aug 26, 2025

lukevalenta mentioned this pull request Aug 26, 2025

mtc: Add option to fetch SPKI from bootstrap entry #97

Closed

bwesterb reviewed Aug 26, 2025

View reviewed changes

lukevalenta commented Aug 26, 2025

View reviewed changes

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

bwesterb reviewed Aug 26, 2025

View reviewed changes

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

Comment thread crates/tlog_tiles/src/tlog.rs

bwesterb reviewed Aug 26, 2025

View reviewed changes

bwesterb requested changes Aug 26, 2025

View reviewed changes

bwesterb self-requested a review August 26, 2025 17:55

bwesterb approved these changes Aug 26, 2025

View reviewed changes

lukevalenta added 15 commits August 27, 2025 14:41

docs: fix linter complaint

da7cfb3

Combine ConsistencyProof and InclusionProof types

b30e0d1

Later, we'll use the same subroutine to generate both types of proofs, so it's useful to merge them into a single type.

Rework API for computing Merkle Tree hashes

1ad60c1

* Add _internal suffix to subtree_hash and subtree_indexes functions. These will get removed later. * Add subtree_hash_indexes, subtree_hash, and tree_hash_indexes functions with better API for public consumption.

Fix test to correctly check consistency proofs

e57cdf5

bugfix: Ensure that MTC validity bound is a valid interval

5b008d0

Remove redundant checks handled by walk_subproof

0b37e26

Rename InvalidInput -> ConditionNotMet

d5140e1

Clean up subtree consistency proof verification logic

aaa2a75

Use language from ietf-plants-wg/merkle-tree-certs#139 which aims to clarify some ambiguities.

Use next_power_of_two function, and address some reviewer comments

c04339b

lukevalenta force-pushed the lvalenta/subtree-proofs branch from bbf8a2e to c04339b Compare August 28, 2025 04:25

lukevalenta added 3 commits August 28, 2025 09:50

Support proofs with empty trees, add more tests for subtree proofs

f6afb41

Document parameters for CheckpointCallbacker

bd77be4

Simplify subtree_for_index based on Bas' suggestion

df8c826

Expand documention on walk_subtree recursion

c0b4750

lukevalenta mentioned this pull request Aug 28, 2025

Add initial log entry during initialization, fixes #100 #102

Merged

rozbb approved these changes Aug 29, 2025

View reviewed changes

lukevalenta added 3 commits August 29, 2025 12:18

Rename serialize/deserialize -> to_bytes/from_bytes

705b9db

Expand comments on CheckpointCallbacker Fn closure

b369c53

cjpatton reviewed Aug 29, 2025

View reviewed changes

Comment thread crates/mtc_api/src/landmark.rs Outdated

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

Comment thread crates/tlog_tiles/src/tlog.rs Outdated

Comment thread crates/generic_log_worker/src/log_ops.rs Outdated

lukevalenta added 2 commits September 2, 2025 17:17

Remove anyhow dependency from mtc_api crate

74584ee

Address Chris' comments

de527e4

* Use debug_assert instead of assert in tlog library * Improve documentation and variable naming for stored hash index * Add link to spec for subtree definition

lukevalenta force-pushed the lvalenta/subtree-proofs branch from c3d0239 to de527e4 Compare September 2, 2025 21:21

lukevalenta merged commit 70d68d0 into main Sep 2, 2025
1 check passed

lukevalenta deleted the lvalenta/subtree-proofs branch September 2, 2025 21:27

Uh oh!

Conversation

lukevalenta commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bwesterb left a comment

Choose a reason for hiding this comment

Uh oh!

bwesterb left a comment

Choose a reason for hiding this comment

Uh oh!

rozbb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjpatton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lukevalenta commented Aug 26, 2025 •

edited

Loading

rozbb left a comment •

edited

Loading