Refactor indexer to improve example_crates interface
#40
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t_pubkey Replaces the `xonly_pubkey` field in the `SpOut` struct with `script_pubkey` to directly store the complete script rather than just the public key. This eliminates redundant script generation in multiple places, as the script was being recalculated from the public key whenever needed. The change simplifies the code flow and reduces unnecessary conversions between public key formats and script generation.
…d_spout_for_tweak Extract the logic for finding silent payment outputs with a given tweak into a separate function `find_spout_for_tweak`. This improves code organization by separating the loop iterating through tweaks from the logic that processes each tweak. The new function takes the necessary context and returns a single matched output if found, allowing the main function to focus on collecting results.
Adds two functions for silent payment PSBT handling: 1. `add_sp_data_to_input` - Adds silent payment data (spend public key and tweak) to a PSBT input as proprietary fields 2. `sign_sp` - Signs silent payment inputs in a PSBT by finding inputs with silent payment data, deriving the appropriate tweaked keys, and adding taproot key signatures These functions enable proper signing of PSBTs spending silent payment outputs.
…eys for PSBT outputs Adds functionality to derive silent payment outputs in PSBTs using silent payment private keys. Introduces a new feature flag `psbt_sp_spend` that enables retrieval of silent payment secrets from PSBT inputs. The implementation checks for proprietary BIP352 fields in PSBT inputs before falling back to existing key derivation methods.
…om TxOut Adds a new SpMeta struct to store just the tweak and label information from Silent Payments outputs, with a conversion implementation from SpOut to SpMeta. This allows separating the transaction output data from its metadata, improving code organization and making it easier to handle Silent Payment metadata independently.
Reorders enumeration and filtering operations when collecting outputs to check in silent payments scanning. The previous code filtered before enumerating, causing incorrect outpoint indices. Now it enumerates first to preserve the original transaction output indices, then filters for P2TR outputs, ensuring correct outpoint construction.
…label_tweak} Add three helper functions to encapsulate hash computation logic used throughout the codebase: - `get_input_hash`: Computes a scalar from smallest outpoint and public key sum - `get_shared_secret`: Derives a secret key from a shared secret and derivation index - `get_label_tweak`: Generates a deterministic scalar from a scan key and numeric label This change removes duplicated hash computation code and centralizes these operations in the hashes module for better code organization and reuse.
Remove outdated comment about OutPoint redundancy in SpIndexer struct that explained implementation details no longer relevant to the current codebase.
…outs Pass a reference to the BTreeMap instead of cloning it in scan_txouts function to avoid unnecessary memory allocation. This improves efficiency by preventing duplication of the label_lookup table during transaction scanning.
Updates `cargo run` commands in the justfile to explicitly specify the example_silentpayments package with the -p flag when running the example_silentpayments binary, ensuring the correct package is targeted during execution.
nymius
force-pushed
the
refactor/indexer
branch
3 times, most recently
from
62c4d88 to
dbf9584
Compare
… struct Create a Label struct to encapsulate the data previously stored in BTreeMap<PublicKey, (Scalar, u32)>. This implementation includes serialization/deserialization support and conversion methods. Additionally, add helper serialization functions for Scalar types and label maps to support persistent storage.
Implements a new `SpIndex` struct that provides functionalities for indexing and accessing silent payment related data. The struct includes methods for indexing labels, partial secrets, and outputs, along with querying capabilities by script, label, transaction, and other parameters. This implementation will serve as a replacement for the existing SpIndexes functionality.
Move the SpIndex struct and related data structures to a new `indexes.rs` module. This reorganization improves code structure by separating index-related functionality from the main library file. No functional changes were made.
Introduce SpPub struct to encapsulate the minimal data needed to scan for silent payments onchain. The struct stores the spend public key and scan secret key, and provides a method to create labels using these keys. This refactoring centralizes the silent payment scanning capabilities by adding a create_label method that generates the necessary cryptographic components for scanning.
Make serde serialization optional via a new "serde" feature flag. This allow smaller builds for code that doesn't use serialization/deserialization.
Introduces a new implementation of the Silent Payments indexer (SpIndexerV2) that replaces the existing SpIndexer. The new implementation refactors and increases the functionality for scanning, indexing, and tracking silent payment transactions, including methods for block processing, transaction relevance determination, and label management. Updates bdk_chain dependency to version 0.23.0 to support the new implementation.
Adds a new wallet crate for managing silent payments with these capabilities: - Wallet creation from descriptor keys - Balance calculation - Transaction management - Address derivation for spending and change - Chain state tracking - Block processing for relevant transactions - Integration with bdk_tx for transaction handling The wallet implementation provides a core structure for handling silent payment transactions with appropriate security measures and Bitcoin protocol compatibility.
Add a new example crate `example_silentpayments_v2` that implements a command-line interface for Silent Payments wallet functionality. This example demonstrates: - Creating and managing Silent Payment wallets - Deriving Silent Payment scripts for transactions - Displaying Silent Payment codes - Scanning the blockchain for Silent Payment outputs via RPC - Creating transactions with both regular and Silent Payment outputs - Checking wallet balances The example serves as a practical reference implementation for integrating the SpWallet library into applications.
Update example_silentpayments dependencies to their latest versions and adjust code to accommodate API changes. This includes: - Upgrading bdk_file_store from 0.19.0 to 0.21.0 - Upgrading bdk_chain from 0.21.1 to 0.23.0 - Upgrading bdk_bitcoind_rpc from 0.18.0 to 0.20.0 - Adding required CanonicalizationParams parameter to balance and filter methods - Updating Emitter constructor with NO_EXPECTED_MEMPOOL_TXIDS parameter - Moving bdk_bitcoind_rpc imports outside the indexer module
Replace custom BDK example binaries with bdk-cli in the justfile scripts. This change: - Switches from the forked BDK repo to official bdk-cli repository - Uses bdk-cli's standardized wallet commands instead of custom example binaries - Simplifies the silent payment flow implementation to use standard tools - Updates the Docker build to include only the bdk-cli binary instead of multiple example binaries - Adds libssl3 dependency required by bdk-cli This change improves maintainability by using official tooling rather than custom examples.
Add a new `oracles` crate that provides: 1. Kyoto-based filter infrastructure for storing, managing, and subscribing to compact block filters 2. BlindBit client implementation that connects to tweak server endpoints 3. Mechanisms to match tweaks against filters to identify transactions of interest This enables wallet implementations to efficiently retrieve and process filter/tweak data from external sources instead of computing them locally, improving performance for silent payment scanning.
This commit adds a new command `ScanCbf` to the example Silent Payments application that allows users to scan for payments using kyoto-cbf and blindbit. The feature: 1. Integrates the `bdk_sp_oracles` dependency to enable communication with kyoto compact block filter nodes 2. Implements a tokio-based asynchronous scanning mechanism to detect Silent Payments via compact block filters 3. Sets up connections to trusted peers and subscribes to filter events 4. Processes matched transactions by verifying them against blindbit tweak server 5. Updates the wallet when relevant payments are found This enhances the example by demonstrating a more efficient way to scan for Silent Payments without requiring a full node RPC connection.
Restructured the example code into a proper CLI component by: - Renaming directory from `example-crates` to `cli` - Reorganizing examples into `cli/v1` and `cli/v2` subdirectories - Renaming packages from `example_silentpayments*` to `bdk_sp_cli_v*` - Adding binary definitions with proper executable names - Updating workspace members in Cargo.toml - Adjusting exclude patterns in justfile This change improves project organization by treating CLI tools as first-class components rather than examples, making the project structure more intuitive and providing clearer naming conventions for both directory structure and package names.
Move the SpIndexerV2 implementation from lib.rs into a dedicated v2 module to improve code organization and maintainability. All existing functionality is preserved while reducing the size of lib.rs. This commit restructures the codebase by moving the SpIndexerV2 implementation and its related code (including indexes.rs) into a dedicated v2 module. The code remains functionally identical, but is now better organized with clearer module boundaries. All import paths in dependent files (oracles/tweaks/blindbit.rs, wallet/lib.rs, wallet/signers.rs) have been updated to reference the new module location.
…_ref_to_slice_refs lint
nymius
force-pushed
the
refactor/indexer
branch
from
dbf9584 to
a3bafbf
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces significant restructuring and upgrades to the Silent Payments examples and related tooling. Key changes include:
Restructuring and renaming of example crates:
example-crates/example_silentpaymentsinto a new CLI workspace structure undercli/v1andcli/v2.sp-cli1,sp-cli2) for better separation of legacy and new CLI workflows.example-cratestoclifor more logical organization.New CLI v2 (
cli/v2) supporting:Dependency upgrades:
bdk_chain,bdk_file_store,bdk_bitcoind_rpc, andbitcoincrate versions across crates (indexer,silentpayments,cli/v1).serdefeature flags to certain crates for lightweight builds.psbt_sp_spendfeature insilentpayments).New workspace members:
wallet: ImplementsSpWallet, handling indexing, chain state, balance tracking, labelled addresses, and PSBT candidate selection.oracles: Provides Kyoto-based block filter and Blindbit-based tweak oracle integration.SilentPayments library improvements:
get_label_tweak,get_input_hash,get_shared_secret).SpOutto store fullscript_pubkeyinstead of just x-only keys.scan_txouts) with separation of finding spouts from tweaks.add_sp_data_to_input,sign_sp).CI / Justfile updates:
bdk_sp_cli_v1,bdk_sp_cli_v2) from defaultcargo checkandcargo testruns.justfileto usebdk-cliinstead of legacy BDK example crates.Example usage (CLI v2)
Here are example snippets using the new v2 CLI (
sp-cli2):Notes to the reviewers
cli/v1andcli/v2was done to separate stable, legacy CLI features from new experimental functionality (wallet,oracles, tweak scanning).wallet,oracles,cli/v2) will help check design consistency.--feature psbt_sp_spend) are behind feature flags to avoid coupling core library users with experimental signing functionality.Changelog notice
psbt_sp_spendfeature).bdk_sp_cli_v2with wallet management, silent payment aware transaction construction, and multiple scanning backends (RPC and Kyoto+Blindbit).bdk_sp_walletcrate for managing wallet state and persistence.bdk_sp_oraclescrate with filter/oracle subscription integrations.example_silentpaymentsintobdk_sp_cli_v1; update dependencies to latestbdk,bdk-chain, andbitcoin.indexerupgraded to v2 model with label + shared secret indexing.example_electrum,example_esplora, etc.) from Docker builds.Checklists
All Submissions:
just p(fmt, clippy and test) before committingNew Features: