Skip to content

Conversation

@eval-exec
Copy link
Collaborator

@eval-exec eval-exec commented Jul 7, 2025

What problem does this PR solve?

Issue Number: close #4921

Problem Summary:

This PR significantly enhances the ckb export and ckb import commands with range selection, stdin/stdout support, parallel processing, and improved performance.

What is changed and how it works?

+ rm -rf /tmp/a
+ rm -rf /tmp/export
+ /tmp/ckb init -C /tmp/a
WARN: Mining feature is disabled because of the lack of the block assembler config options.
Initialized CKB directory in /tmp/a
Create ckb.toml
Create ckb-miner.toml
Create default.db-options
Genesis Hash: 0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5
+ /tmp/ckb export -C /home/exec/box/ckb_data --target /tmp/export --from 1 --to 200
Writing to /tmp/export/ckb-1-200.jsonl
[00:00:00] ##################################################    201/201    done!
+ ls -ahl /tmp/export
total 1.1M
drwxr-xr-x   2 exec users   60 Jul 22 15:05 .
drwxrwxrwt 109 root root  2.7K Jul 22 15:05 ..
-rw-r--r--   1 exec users 1.1M Jul 22 15:05 ckb-1-200.jsonl
+ /tmp/ckb export -C /home/exec/box/ckb_data --target /tmp/export --from 200 --to 300
Writing to /tmp/export/ckb-200-300.jsonl
[00:00:00] ##################################################    301/301    done!
+ ls -ahl /tmp/export
total 3.3M
drwxr-xr-x   2 exec users   80 Jul 22 15:05 .
drwxrwxrwt 109 root root  2.7K Jul 22 15:05 ..
-rw-r--r--   1 exec users 1.1M Jul 22 15:05 ckb-1-200.jsonl
-rw-r--r--   1 exec users 2.3M Jul 22 15:05 ckb-200-300.jsonl
+ /tmp/ckb import -C /tmp/a /tmp/export/ckb-200-300.jsonl
Before import, current tip is 0-Byte32(0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5)
Import error: Custom { kind: Other, error: "In /tmp/export/ckb-200-300.jsonl, the first block is 200-Byte32(0xd7f0a6328a466724a2edabc7ba990bf27a092ec3b555b8d67bd8c9cff0fa6eff), and its parent (hash: Byte32(0x5c75b4fc749d23238a1999296c27359b28043578af84fe0e11f126b75b1d8e53)) was not found in the database. The current tip is 0-Byte32(0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5)." }
+ true
+ /tmp/ckb import -C /tmp/a /tmp/export/ckb-1-200.jsonl
Before import, current tip is 0-Byte32(0x92b197aa1fba0f63633922c61c92375c9c074a93e85963554f5499fe1450d0e5)
[00:00:00] ################################################## 1.03MiB/1.03MiB done!
After import, Current tip is 200-Byte32(0xd7f0a6328a466724a2edabc7ba990bf27a092ec3b555b8d67bd8c9cff0fa6eff)
+ /tmp/ckb import -C /tmp/a /tmp/export/ckb-200-300.jsonl
Before import, current tip is 200-Byte32(0xd7f0a6328a466724a2edabc7ba990bf27a092ec3b555b8d67bd8c9cff0fa6eff)
[00:00:01] ################################################## 2.22MiB/2.22MiB done!
After import, Current tip is 300-Byte32(0x2461b3c944f041db2fcf414b0368abae64aa7d72365709bfdd02a41f8ca2a8c2)

Related changes

  • Support range feature for ckb export
  • Support verifier selection for ckb import
  • Export to stdout
  • Import from stdin
  • Todo Benchmark

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code ci-runs-only: [ quick_checks,linters ]

Side effects

  • None

Release note

Note:

  ### Breaking Changes
  ⚠️ **Export format changed from `.json` to `.jsonl`** (JSON Lines format)

  ### New Features

  **Export:**
  - `--from` and `--to` for block range selection (both inclusive)
  - Export to stdout: `ckb export -t -` (enables piping)
  - Parallel processing for better performance

  **Import:**
  - `--num-threads N` to control parallel processing
  - Import from stdin: `ckb import -` (enables piping)
  - `--skip-script-verify` and `--skip-all-verify` flags for faster trusted imports


@eval-exec eval-exec requested a review from a team as a code owner July 7, 2025 07:12
@eval-exec eval-exec requested review from doitian and removed request for a team July 7, 2025 07:12
@eval-exec eval-exec force-pushed the exec/enhance-import-export branch 6 times, most recently from 627c4cf to bbf2420 Compare July 10, 2025 06:16
@eval-exec eval-exec force-pushed the exec/enhance-import-export branch 4 times, most recently from a409e5d to 06d221e Compare July 22, 2025 09:47
@eval-exec eval-exec force-pushed the exec/enhance-import-export branch 7 times, most recently from d3ffd22 to 4a627eb Compare July 31, 2025 06:21
@eval-exec eval-exec force-pushed the exec/enhance-import-export branch 3 times, most recently from 6ea7b59 to ce6dc5b Compare August 7, 2025 04:08
Comment on lines 167 to 176
batch.par_iter().for_each(|line| {
let block: JsonBlock =
serde_json::from_str(line).expect("parse block from json");
let block: Arc<core::BlockView> = Arc::new(block.into());
blocks_tx.send(block).expect("send block to channel");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serde-rs/json#635 Need to limit the number of parallel serde json, its memory usage is very large

Copy link
Collaborator Author

@eval-exec eval-exec Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory usage is bounded by num_threads (defaults to CPU cores).

CKB max block size is 0.6M, on a 128-core systems, I think worst case is about 0.6MB * 10x serde overhead * 128 cores = 770MB is acceptable. Users can lower --num-threads if needed.

@eval-exec eval-exec force-pushed the exec/enhance-import-export branch 6 times, most recently from d4012b3 to 9393345 Compare August 14, 2025 06:25
@eval-exec eval-exec force-pushed the exec/enhance-import-export branch from 25501d4 to 832e33c Compare August 20, 2025 06:42
driftluo
driftluo previously approved these changes Aug 25, 2025
@eval-exec eval-exec force-pushed the exec/enhance-import-export branch 2 times, most recently from 4c185c5 to 3d6e6c6 Compare December 16, 2025 03:17
.long(ARG_TO)
.value_name("to")
.required(false)
.help("Specify the to block number/hash for export"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear whether to is inclusive or exclusive.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated:

        .arg(
            Arg::new(ARG_FROM)
                .long(ARG_FROM)
                .value_name("from")
                .required(false)
                .help("Specify the starting block number/hash for export (inclusive)"),
        )
        .arg(
            Arg::new(ARG_TO)
                .long(ARG_TO)
                .value_name("to")
                .required(false)
                .help("Specify the ending block number/hash for export (inclusive)"),

@eval-exec eval-exec force-pushed the exec/enhance-import-export branch from 3d6e6c6 to 59f5610 Compare January 6, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🏗 In progress

Development

Successfully merging this pull request may close these issues.

Enhance ckb export/import for flexible ranges, faster import, and JSONL output

3 participants