feat(iota-light-client)!: extend light client to use checkpoint archive (#6135)

# Description of change

Porting:
~~MystenLabs/sui#19028 has already been merged
to `develop` separatedly, but this PR started earlier and also included
it.
MystenLabs/sui#19604
MystenLabs/sui#20786

This adds the following features:
[x] better crate structure
[x] syncing from an archive store
[x] syncing from a checkpoint store

The following changes and features were added on top:
[x] syncing from full node only (useful for local networks and
non-pruning full nodes and as a last resort fallback)
[x] automatic setup including downloading the genesis.blob
[x] GraphQL query through `iota-graphql-rpc-client` instead of bare
metal `reqwest` as in upstream
[x] The old testdata has been removed and replaced and stored in a
`fixtures` directory
[x] the light client CLI resides in `main.rs` again, because it's more
consistent with other binaries
[x] improved error handling: user-facing panics have been replaced by
error messages
[x] improved configuration of the light-client { automatic sync,
automatic genesis.blob download }
[x] switched to new historical checkpoint endpoint for devnet
`checkpoints.devnet.iota.cafe`
[x] fixed a lot minor upstream issues
[x] no upfront setup necessary to use the light client, which is
different upstream

## Links to any relevant issues

Partly fixes #5368 . In particular adds support for syncing the light
client from a checkpoint archive, e.g. S3 or local directory, a remote
checkpoint summary object store, or by directly querying a full node
using its RPC and GraphQL endpoints.

Also fixes #6090 

## Type of change

Choose a type of change, and delete any options that are not relevant.

- Bug fix (a non-breaking change which fixes an issue)
- Enhancement (a non-breaking change which adds functionality)
- Breaking change (fix or feature that would cause existing
functionality to not work as expected)


## How the change has been tested

- Unit tests
- devnet: full support of { Sync, Check-* } commands
- 
```
cargo run --bin iota-light-client -- --config devnet.yaml sync
cargo run --bin iota-light-client -- --config devnet.yaml check-transaction HAEPr4LGQXbUQzANTpKcK6eBFaGtztc1noh8jw1mPCKG
cargo run --bin iota-light-client -- --config devnet.yaml check-object 0x80eb8f6e548d558e8d71d85688cf3a82186c6763cb8a499c387694323a60abdf
```
- testnet: full support of { Sync } command; `Check-` commands can
currently only be tested on `devnet` as the corresponding `testnet`
endpoint for historical checkpoints is not available yet

- localnet full support of { Sync, Check-* } commands with this config:
```yaml
rpc_url: "http://localhost:9000"
graphql_url: "http://localhost:9125"
checkpoints_dir: "checkpoints_localnet"
genesis_blob_download_url: "file:///home/me/vps/.iota/iota_config/genesis.blob"
sync_before_check: true
object_store_url: ~
archive_store_config: ~
```

## Change checklist

Tick the boxes that are relevant to your changes, and delete any items
that are not.

- [x] I have followed the contribution guidelines for this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have checked that new and existing unit tests pass locally with
my changes

### Release Notes

<!--
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
-->

- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: George Danezis <[email protected]>
Co-authored-by: Sadhan Sood <[email protected]>
Co-authored-by: Deepak Maram <[email protected]>
Co-authored-by: Thoralf-M <[email protected]>
Co-authored-by: DaughterOfMars <[email protected]>
Co-authored-by: Thibault Martinez <[email protected]>

Author: 7 people

Cargo.lock

@@ -265,6 +265,7 @@ fdlimit = "0.2.1"
 flate2 = "1.0.28"
 fs_extra = "1.3.0"
 futures = "0.3.28"
+getset = "0.1"
 git-version = "0.3.5"
 governor = "0.6.0"
 hdrhistogram = "7.5.1"

Cargo.toml

@@ -210,6 +210,35 @@ impl Manifest {
             }
         }
     }
+
+    pub fn get_all_end_of_epoch_checkpoint_seq_numbers(&self) -> Result<Vec<u64>> {
+        match self {
+            Manifest::V1(manifest) => {
+                let mut summary_files: Vec<_> = manifest
+                    .file_metadata
+                    .clone()
+                    .into_iter()
+                    .filter(|f| f.file_type == FileType::CheckpointSummary)
+                    .collect();
+                summary_files.sort_by_key(|f| f.checkpoint_seq_range.start);
+                assert_eq!(summary_files.first().unwrap().checkpoint_seq_range.start, 0);
+                // get last checkpoint seq num per epoch
+                let res = summary_files.windows(2).filter_map(|w| {
+                    assert_eq!(
+                        w[1].checkpoint_seq_range.start,
+                        w[0].checkpoint_seq_range.end
+                    );
+                    if w[1].epoch_num == w[0].epoch_num + 1 {
+                        Some(w[0].checkpoint_seq_range.end - 1)
+                    } else {
+                        None
+                    }
+                });
+                Ok(res.collect())
+            }
+        }
+    }
+
     pub fn update(
         &mut self,
         epoch_num: u64,

crates/iota-archival/src/lib.rs

@@ -336,13 +336,13 @@ impl ArchiveReader {
     pub async fn read_summaries_for_list_no_verify<S>(
         &self,
         store: S,
-        skiplist: Vec<CheckpointSequenceNumber>,
+        checkpoints: Vec<CheckpointSequenceNumber>,
         checkpoint_counter: Arc<AtomicU64>,
     ) -> Result<()>
     where
         S: WriteStore + Clone,
     {
-        let summary_files = self.get_summary_files_for_list(skiplist.clone()).await?;
+        let summary_files = self.get_summary_files_for_list(checkpoints.clone()).await?;
         let remote_object_store = self.remote_object_store.clone();
         let stream = futures::stream::iter(summary_files.iter())
             .map(|summary_metadata| {
@@ -365,7 +365,7 @@ impl ArchiveReader {
                     )
                     .and_then(|summary_iter| {
                         summary_iter
-                            .filter(|s| skiplist.contains(&s.sequence_number))
+                            .filter(|s| checkpoints.contains(&s.sequence_number))
                             .try_for_each(|summary| {
                                 Self::insert_certified_checkpoint(&store, summary)?;
                                 checkpoint_counter.fetch_add(1, Ordering::Relaxed);

crates/iota-archival/src/reader.rs

@@ -1 +1,4 @@
-checkpoints_dir/*
+checkpoints_mainnet/
+checkpoints_testnet/
+checkpoints_devnet/
+checkpoints_localnet/

crates/iota-light-client/.gitignore

@@ -5,13 +5,18 @@ authors = ["IOTA Foundation <[email protected]>"]
 edition = "2021"
 license = "Apache-2.0"
 publish = false
+description = "A light client for the IOTA blockchain"
 
 [lib]
 path = "src/lib.rs"
 
 [[bin]]
 name = "iota-light-client"
-path = "src/bin/light_client.rs"
+path = "src/main.rs"
+
+[[bin]]
+name = "generate-checkpoint-snapshots"
+path = "src/bin/generate_checkpoint_snapshots.rs"
 
 [dependencies]
 # external dependencies
@@ -20,20 +25,33 @@ async-trait.workspace = true
 bcs.workspace = true
 clap.workspace = true
 env_logger = "0.11.5"
+futures.workspace = true
+getset.workspace = true
 log = "0.4.22"
 object_store.workspace = true
+prometheus.workspace = true
 reqwest.workspace = true
+roaring.workspace = true
 serde.workspace = true
 serde_json.workspace = true
 serde_yaml.workspace = true
 tokio = { workspace = true, features = ["full"] }
+tracing.workspace = true
 url.workspace = true
 
 # internal dependencies
+bin-version.workspace = true
+iota-archival.workspace = true
 iota-config.workspace = true
+iota-data-ingestion-core.workspace = true
+iota-graphql-rpc-client.workspace = true
 iota-json-rpc-types.workspace = true
 iota-package-resolver.workspace = true
 iota-rest-api.workspace = true
 iota-sdk.workspace = true
 iota-types.workspace = true
 move-core-types.workspace = true
+telemetry-subscribers.workspace = true
+
+[dev-dependencies]
+tempfile.workspace = true

crates/iota-light-client/Cargo.toml

@@ -20,45 +20,77 @@ The light client requires a config file and a directory to cache checkpoints, an
 
 ## Setup
 
-The config file for the light client takes a URL for a full node, a directory (that must exist) and within the directory to name of the genesis blob for the IOTA network.
+The config file for the light client takes a URL for a full node, a directory to store checkpoint summaries (that must exist) and within the directory the name of the genesis blob for the IOTA network.
 
 ```
-full_node_url: "https://api.testnet.iota.cafe:443"
-checkpoint_summary_dir: "checkpoints_dir"
-genesis_filename: "genesis.blob"
+# A full node JSON RPC endpoint to query the latest network state (mandatory)
+rpc_url: "https://api.mainnet.iota.cafe"
+
+# A full node GraphQL RPC endpoint to query end-of-epoch checkpoints (optional if archive store config is provided)
+graphql_url: "https://graphql.mainnet.iota.cafe"
+
+# Local directory to store checkpoint summaries and other synchronization data (mandatory) 
+checkpoints_dir: "checkpoints_mainnet"
+
+# A URL to download or copy the genesis blob file from (optional if genesis blob is already present in checkpoints_dir)
+genesis_blob_download_url: "https://dbfiles.mainnet.iota.cafe/genesis.blob"
+
+# A flag to set whether the light client should always sync before checking an object or a transaction for inclusion (mandatory)
+sync_before_check: true
+
+# A config for an object store that gets populated by a historical checkpoint writer (optional)
+checkpoint_store_config:
+  object-store: "S3"
+  aws-endpoint: "https://checkpoints.mainnet.iota.cafe"
+  aws-virtual-hosted-style-request: true
+  no-sign-request: true
+  aws-region: "weur"
+  object-store-connection-limit: 20
+
+# A config for an object store that gets populated by an archiver (optional)
+archive_store_config:
+  object-store: "S3"
+  aws-endpoint: "https://archive.mainnet.iota.cafe"
+  aws-virtual-hosted-style-request: true
+  no-sign-request: true
+  aws-region: "weur"
+  object-store-connection-limit: 20
 ```
 
-The genesis blob for the IOTA mainnet can be found here: https://github.com/iotaledger/iota-genesis/blob/main/mainnet/genesis.blob
-
 ## Sync
 
 Every day there is a need to download new checkpoints through sync by doing:
 
 ```
-$ iota-light-client --config light_client.yaml sync
+$ iota-light-client --config testnet.yaml sync
 ```
 
-Where `light_client.yaml` is the config file above.
+Where `testnet.yaml` is the config file above.
 
 This command will download all end-of-epoch checkpoints, and check them for validity. They will be cached within the checkpoint summary directory for use by future invocations.
 
+Internally, sync works in two steps. It first downloads the end-of-epoch checkpoint numbers into the `checkpoints.yaml` file (which needs to be present in the checkpoint summaries directory). Next, it downloads the corresponding checkpoint summaries.
+
 ## Check Transaction
 
-To check a transaction was executed, as well as the events it emitted do:
+To check whether a transaction was executed in the testnet and its effects and events exist, run:
 
 ```
-$ iota-light-client --config light_client.yaml transaction -t 8RiKBwuAbtu8zNCtz8SrcfHyEUzto6zi6cMVA9t4WhWk
+$ iota-light-client --config testnet.yaml check-transaction <transaction-digest>
 ```
 
-Where the base58 encoding of the transaction ID is specified. If the transaction has been executed the transaction ID the effects digest are displayed and all the events are printed in JSON. If not an error is printed.
+where `transaction-digest` is a base58 encoded string. If the transaction has been executed in the past, its digest, the effects, and all events are displayed. Events are printed in JSON. Otherwise an error is shown.
+
+If you set `sync_before_check: true` in the config, the light client will first sync itself to the latest network state before checking the transaction.
 
 ## Check Object
 
-To check an object provide its ID in the following way:
+To check whether an object exists in the testnet, run:
 
 ```
-$ iota-light-client --config light_client.yaml object -o 0xc646887891adfc0540ec271fd0203603fb4c841a119ec1e00c469441
-abfc7078
+$ iota-light-client --config testnet.yaml check-object <object-id>
 ```
 
-The object ID is represented in Hex as displayed in explorers. If the object exists in the latest state it is printed out in JSON, otherwise an error is printed.
+where `object-id` is a hex encoded string with a 0x prefix. If the object exists, it is printed in JSON. Otherwise an error is shown.
+
+If you set `sync_before_check: true` in the config, the light client will first sync itself to the latest network state before checking the object.

crates/iota-light-client/README.md

@@ -0,0 +1,19 @@
+rpc_url: "https://api.devnet.iota.cafe"
+graphql_url: "https://graphql.devnet.iota.cafe"
+checkpoints_dir: "checkpoints_devnet"
+genesis_blob_download_url: "https://dbfiles.devnet.iota.cafe/genesis.blob"
+sync_before_check: true
+checkpoint_store_config:
+  object-store: "S3"
+  aws-endpoint: "https://checkpoints.devnet.iota.cafe"
+  aws-virtual-hosted-style-request: true
+  no-sign-request: true
+  aws-region: "weur"
+  object-store-connection-limit: 20
+archive_store_config:
+  object-store: "S3"
+  aws-endpoint: "https://archive.devnet.iota.cafe"
+  aws-virtual-hosted-style-request: true
+  no-sign-request: true
+  aws-region: "weur"
+  object-store-connection-limit: 20