Introduce graph sync crate for fast-forwarding through gossip data downloaded from a server.

Author: arik-so

.github/workflows/build.yml

@@ -138,6 +138,7 @@ jobs:
         run: |
           cargo test --verbose --color always  -p lightning
           cargo test --verbose --color always  -p lightning-invoice
+          cargo test --verbose --color always -p lightning-graph-sync
           cargo build --verbose  --color always -p lightning-persister
           cargo build --verbose  --color always -p lightning-background-processor
       - name: Test C Bindings Modifications on Rust ${{ matrix.toolchain }}

Cargo.toml

@@ -7,6 +7,7 @@ members = [
     "lightning-net-tokio",
     "lightning-persister",
     "lightning-background-processor",
+    "lightning-graph-sync"
 ]
 
 exclude = [

lightning-graph-sync/Cargo.toml

@@ -0,0 +1,16 @@
+[package]
+name = "lightning-graph-sync"
+version = "0.0.104"
+authors = ["Arik Sosman <[email protected]>"]
+license = "MIT OR Apache-2.0"
+repository = "https://github.com/lightningdevkit/rust-lightning"
+edition = "2018"
+description = """
+Utility to fetch gossip routing data from LNSync or LNSync-like server
+"""
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+lightning = { version = "0.0.105", path = "../lightning" }
+bitcoin = { version = "0.27", default-features = false, features = ["secp-recovery"] }

lightning-graph-sync/README.md

@@ -0,0 +1,91 @@
+# lightning-graph-sync
+
+This crate exposes functionality for rapid gossip graph syncing, aimed primarily at mobile clients.
+
+## Mechanism
+
+The (presumed) server sends a compressed gossip response containing gossip data. The gossip data is formatted compactly,
+omitting signatures and opportunistically incremental.
+
+Essentially, the serialization structure is as follows:
+
+1. Fixed prefix bytes `76, 68, 75, 1` (the first three bytes are ASCII for `LDK`)
+    - The purpose of this prefix is to identify the serialization format, should other rapid gossip sync formats arise
+      in the future.
+    - The fourth byte is the protocol version in case our format gets updated
+2. Chain hash (32 bytes)
+3. Latest seen timestamp (`u32`)
+4. A `u64` indicating the number of node IDs to follow
+5. `[PublicKey]` (array of compressed 33-byte node IDs)
+6. A `u64` indicating the number of channel announcement messages to follow
+7. `[CustomChannelAnnouncement]` (array of significantly stripped down channel announcements)
+8. A `u64` indicating the number of channel update messages to follow
+9. A `u8` flagging whether non-incremental updates are present (if yes, it's set to `1`)
+    - If present, the following values are added:
+        1. `default_cltv_expiry_delta`: `u16`
+        2. `default_htlc_minimum_msat`: `u64`
+        3. `default_fee_base_msat`: `u32`
+        4. `default_fee_proportional_millionths`: `u32`
+        5. `default_htlc_maximum_msat`: `u64` (if the default is no maximum, `u64::MAX`)
+    - The defaults are calculated by the server based on the frequency within non-incremental updates within this
+      particular message
+10. `[CustomChannelUpdate]`
+
+You will also notice that `NodeAnnouncement` messages are skipped altogether.
+
+The data is then applied to the current network graph, artifically backdated 7 days from the current time to make sure
+more recent updates obtained directly from gossip are not accidentally overwritten.
+
+### CustomChannelAnnouncements
+
+To achieve compactness and avoid data repetition, we're sending a significantly stripped down version of the channel
+announcement message, which contains only the following data:
+
+1. `channel_features`: `u16` + `n`, where `n` is the number of bytes indicated by the first `u16`
+2. `short_channel_id`: `CompactSize` (incremental `CompactSize` deltas starting from 0)
+3. `node_id_1_index`: `CompactSize` (index of node id within the previously sent sequence)
+4. `node_id_2_index`: `CompactSize` (index of node id within the previously sent sequence)
+
+### CustomChannelUpdate
+
+For the purpose of rapid syncing, we have deviated from the channel update format specified in BOLT 7 significantly. Our
+custom channel updates are structured as follows:
+
+1. `short_channel_id`: `CompactSize` (incremental `CompactSize` deltas starting at 0)
+2. `custom_channel_flags`: `u8`
+3. `update_data`
+
+Specifically, our custom channel flags break down like this:
+
+| 128                 | 64 | 32 | 16 | 8 | 4 | 2                | 1         |
+|---------------------|----|----|----|---|---|------------------|-----------|
+| Incremental update? |    |    |    |   |   | Disable channel? | Direction |
+
+If the most significant bit is set to `1`, indicating an incremental update, the intermediate bit flags assume the
+following meaning:
+
+| 64                              | 32                              | 16                          | 8                                         | 4                               |
+|---------------------------------|---------------------------------|-----------------------------|-------------------------------------------|---------------------------------|
+| `cltv_expiry_delta` has changed | `htlc_minimum_msat` has changed | `fee_base_msat` has changed | `fee_proportional_millionths` has changed | `htlc_maximum_msat` has changed |
+
+If the most significant bit is set to `0`, the meaning is almost identical, except instead of a change, the flags now
+represent a deviation from the defaults sent at the beginning of the update sequence.
+
+In both cases, `update_data` only contains the fields that are indicated by the channel flags to be non-standard or to
+have changed.
+
+## Delta Calculation
+
+The way a server is meant to calculate this rapid gossip sync data is by using two data points as a reference that are
+meant to be provided by the client:
+`latest_announcement_blockheight` and `latest_update_timestamp`.
+
+Based on `latest_announcement_blockheight`, the server only sends channel announcements that occurred at or after that
+block height.
+
+Based on `latest_update_timestamp`, the server fetches all channel updates that occurred at or after the timestamp.
+Then, the server also checks for each update whether there had been a previous one prior to the given timestamp.
+
+If a particular channel update had never occurred before, the full update is sent. If an channel has had updates prior
+to the provided timestamp, the latest update prior to the timestamp is taken as a reference, and the delta is calculated
+against it.

lightning-graph-sync/src/error.rs

@@ -0,0 +1,33 @@
+use lightning::ln::msgs::{DecodeError, LightningError};
+
+/// All-encompassing standard error type that processing can return
+pub enum GraphSyncError {
+	/// IO error wrapper, typically the result of an issue with the file system
+	IOError(std::io::Error),
+	/// Error trying to read the update data, typically due to an erroneous data length indication
+	/// that is greater than the actual amount of data provided
+	DecodeError(DecodeError),
+	/// Error applying the patch to the network graph, usually the result of updates that are too
+	/// old or missing prerequisite data to the application of updates out of order
+	LightningError(LightningError),
+	/// Some other error whose nature is indicated in its descriptor string
+	ProcessingError(String),
+}
+
+impl From<std::io::Error> for GraphSyncError {
+	fn from(error: std::io::Error) -> Self {
+		Self::IOError(error)
+	}
+}
+
+impl From<DecodeError> for GraphSyncError {
+	fn from(error: DecodeError) -> Self {
+		Self::DecodeError(error)
+	}
+}
+
+impl From<LightningError> for GraphSyncError {
+	fn from(error: LightningError) -> Self {
+		Self::LightningError(error)
+	}
+}

lightning-graph-sync/src/lib.rs

@@ -0,0 +1,105 @@
+#![deny(missing_docs)]
+#![deny(unsafe_code)]
+#![deny(broken_intra_doc_links)]
+#![deny(non_upper_case_globals)]
+#![deny(non_camel_case_types)]
+#![deny(non_snake_case)]
+#![deny(unused_mut)]
+#![deny(unused_variables)]
+#![deny(unused_imports)]
+
+//! This crate exposes functionality to rapidly sync gossip data, aimed primarily at mobile
+//! devices.
+
+use std::fs::File;
+
+use lightning::routing::network_graph;
+
+use crate::error::GraphSyncError;
+
+/// Error types that these functions can return
+pub mod error;
+
+/// Core functionality of this crate
+pub mod processing;
+
+/// Sync gossip data from a file
+///
+/// `network_graph`: The network graph to apply the updates to
+///
+/// `sync_path`: Path to the file where the gossip update data is located
+///
+pub fn sync_network_graph_with_file_path(
+	network_graph: &network_graph::NetworkGraph,
+	sync_path: &str,
+) -> Result<(), GraphSyncError> {
+	let mut file = File::open(sync_path)?;
+	processing::read_network_graph(&network_graph, &mut file)
+}
+
+#[cfg(test)]
+mod tests {
+	use std::fs;
+
+	use bitcoin::blockdata::constants::genesis_block;
+	use bitcoin::Network;
+
+	use lightning::routing::network_graph::NetworkGraph;
+
+	use crate::sync_network_graph_with_file_path;
+
+	#[test]
+	fn test_sync_from_file() {
+		// same as incremental_only_update_fails_without_prior_same_direction_updates
+		// (OldestDataDirection1, timestamp 0)
+		let valid_response = vec![
+			76, 68, 75, 1, 111, 226, 140, 10, 182, 241, 179, 114, 193, 166, 162, 70, 174, 99, 247,
+			79, 147, 30, 131, 101, 225, 90, 8, 156, 104, 214, 25, 0, 0, 0, 0, 0, 97, 227, 98, 218,
+			0, 0, 0, 4, 2, 22, 7, 207, 206, 25, 164, 197, 231, 230, 231, 56, 102, 61, 250, 251,
+			187, 172, 38, 46, 79, 247, 108, 44, 155, 48, 219, 238, 252, 53, 192, 6, 67, 2, 36, 125,
+			157, 176, 223, 175, 234, 116, 94, 248, 201, 225, 97, 235, 50, 47, 115, 172, 63, 136,
+			88, 216, 115, 11, 111, 217, 114, 84, 116, 124, 231, 107, 2, 158, 1, 242, 121, 152, 106,
+			204, 131, 186, 35, 93, 70, 216, 10, 237, 224, 183, 89, 95, 65, 3, 83, 185, 58, 138,
+			181, 64, 187, 103, 127, 68, 50, 2, 201, 19, 17, 138, 136, 149, 185, 226, 156, 137, 175,
+			110, 32, 237, 0, 217, 90, 31, 100, 228, 149, 46, 219, 175, 168, 77, 4, 143, 38, 128,
+			76, 97, 0, 0, 0, 2, 0, 0, 255, 8, 153, 192, 0, 2, 27, 0, 0, 0, 1, 0, 0, 255, 2, 68,
+			226, 0, 6, 11, 0, 1, 2, 3, 0, 0, 0, 2, 1, 0, 40, 0, 0, 0, 0, 0, 0, 3, 232, 0, 0, 3,
+			232, 0, 0, 0, 1, 0, 0, 0, 0, 29, 129, 25, 192, 255, 8, 153, 192, 0, 2, 27, 0, 0, 29, 0,
+			0, 0, 1, 0, 0, 0, 125, 0, 0, 0, 0, 58, 85, 116, 216, 255, 2, 68, 226, 0, 6, 11, 0, 1,
+			1,
+		];
+
+		fs::create_dir_all("./tmp/graph-sync-tests").unwrap();
+		fs::write("./tmp/graph-sync-tests/test_data.lngossip", valid_response).unwrap();
+
+		let block_hash = genesis_block(Network::Bitcoin).block_hash();
+		let network_graph = NetworkGraph::new(block_hash);
+
+		let before = network_graph.to_string();
+		assert_eq!(before.len(), 31);
+
+		let sync_result = sync_network_graph_with_file_path(
+			&network_graph,
+			"./tmp/graph-sync-tests/test_data.lngossip",
+		);
+
+		assert!(sync_result.is_ok());
+
+		let after = network_graph.to_string();
+		assert_eq!(after.len(), 1727);
+		assert!(
+			after.contains("021607cfce19a4c5e7e6e738663dfafbbbac262e4ff76c2c9b30dbeefc35c00643:")
+		);
+		assert!(
+			after.contains("02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b:")
+		);
+		assert!(
+			after.contains("029e01f279986acc83ba235d46d80aede0b7595f410353b93a8ab540bb677f4432:")
+		);
+		assert!(
+			after.contains("02c913118a8895b9e29c89af6e20ed00d95a1f64e4952edbafa84d048f26804c61:")
+		);
+		assert!(after.contains("channels: [619737530008010752]"));
+		assert!(after.contains("channels: [783241506229452801]"));
+	}
+}