-
Notifications
You must be signed in to change notification settings - Fork 246
Enhance ckb export/import subcommand with range and verifier selection
#4924
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Enhance ckb export/import subcommand with range and verifier selection
#4924
Conversation
627c4cf to
bbf2420
Compare
a409e5d to
06d221e
Compare
d3ffd22 to
4a627eb
Compare
6ea7b59 to
ce6dc5b
Compare
util/instrument/src/import.rs
Outdated
| batch.par_iter().for_each(|line| { | ||
| let block: JsonBlock = | ||
| serde_json::from_str(line).expect("parse block from json"); | ||
| let block: Arc<core::BlockView> = Arc::new(block.into()); | ||
| blocks_tx.send(block).expect("send block to channel"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
serde-rs/json#635 Need to limit the number of parallel serde json, its memory usage is very large
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Memory usage is bounded by num_threads (defaults to CPU cores).
CKB max block size is 0.6M, on a 128-core systems, I think worst case is about 0.6MB * 10x serde overhead * 128 cores = 770MB is acceptable. Users can lower --num-threads if needed.
d4012b3 to
9393345
Compare
25501d4 to
832e33c
Compare
4c185c5 to
3d6e6c6
Compare
ckb-bin/src/cli.rs
Outdated
| .long(ARG_TO) | ||
| .value_name("to") | ||
| .required(false) | ||
| .help("Specify the to block number/hash for export"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not clear whether to is inclusive or exclusive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated:
.arg(
Arg::new(ARG_FROM)
.long(ARG_FROM)
.value_name("from")
.required(false)
.help("Specify the starting block number/hash for export (inclusive)"),
)
.arg(
Arg::new(ARG_TO)
.long(ARG_TO)
.value_name("to")
.required(false)
.help("Specify the ending block number/hash for export (inclusive)"),Signed-off-by: Eval EXEC <[email protected]>
Signed-off-by: Eval EXEC <[email protected]>
Signed-off-by: Eval EXEC <[email protected]>
Signed-off-by: Eval EXEC <[email protected]>
Signed-off-by: Eval EXEC <[email protected]>
Signed-off-by: Eval EXEC <[email protected]>
Signed-off-by: Eval EXEC <[email protected]>
Signed-off-by: Eval EXEC <[email protected]>
3d6e6c6 to
59f5610
Compare
59f5610 to
9771297
Compare
What problem does this PR solve?
Issue Number: close #4921
Problem Summary:
This PR significantly enhances the
ckb exportandckb importcommands with range selection, stdin/stdout support, parallel processing, and improved performance.What is changed and how it works?
Related changes
ckb exportckb importCheck List
Tests
Side effects
Release note