Storacha migration tool

[LexLuthr]

⚠️ DO NOT MERGE INTO MAIN

Summary

import-pieces

Summary

Adds a one-off curio import-pieces tool for the Storacha migration.
The tool imports existing Storacha CAR files into Curio piece-park storage by:

• reading source files named as piece CID v2: .car
• moving each file through /storacha-staging
• creating or reusing the matching parked_pieces row
• creating exactly one parked_piece_refs row with data_url = 'storacha-migration'
• creating exactly one matching pdp_piecerefs row for the public PDP service
• moving the file to Curio’s piece-park final path: /piece/s-t00-<parked_piece_id>
• declaring the piece in sector_location
• marking parked_pieces.complete = true

The migration is retryable across the important crash points:

• files already in storacha-staging are retried first
• rows where the target/piece/s-t00- file already exists but complete=false are repaired
• existing live ParkPiece tasks are left alone
• duplicate Storacha refs or mismatched PDP refs fail instead of creating more rows
• Adds coverage for the migration state machine, including fresh imports, staged retries, final-file recovery, live/stale task handling, ref idempotency, duplicate/mismatched ref errors, batch ordering, and non-flaky filesystem error cases.

How To Use

1. Source directory must contain Storacha CAR files named:

   <piece-cid-v2>.car

2. Target must be the directory of an attached Curio storage path, for example:

   /path/to/curio-storage

3. Run in batches:

   curio toolbox import-pieces \
     --source /path/to/storacha-cars \
     --target /path/to/curio-storage \
     --result /path/to/result.json \
     --batch-size 20

   The command writes JSON to --result:

   {
     "count": 20,
     "pieces": ["<piece-cid-v2>", "..."],
     "error": "optional error string on failed runs"
   }

4. After each invocation, read --result even if the command exits non-zero. The file contains the pieces successfully imported by that invocation only. It is not cumulative and may be overwritten by the next run.

5. Re-run until the command exits successfully with count = 0. If a run returns an error, process any pieces present in --result first, then retry or stop based on the error.

aggregate-pieces

Summary

Adds a one-off curio toolbox aggregate-pieces tool for the Storacha migration.

The tool aggregates already-imported Storacha pieces into larger Curio piece-park pieces by:

• reading an input file containing one PieceCIDv2 per line
• treating the input file and source piece storage as read-only
• hashing the input file and storing deterministic resume state under <target>/storacha-aggregate-work/<input-sha256>
• hydrating PieceCIDv2 values from existing complete, long-term parked_pieces rows
• reading source subpieces from source/piece/s-t00-<parked_piece_id>
• bucketing hydrated records by padded piece size
• deduping each bucket with OS sort
• writing one atomic groups.jsonl replay plan
• greedily packing aggregate groups largest-size-first up to 1 GiB padded aggregate size
• validating each group with datasegment before committing the grouping plan
• creating aggregate files directly instead of using task_aggregation
• computing CommP while writing the aggregate file
• staging the verified aggregate file under <target>/storacha-aggregate-staging
• importing the staged aggregate through the same piece-park path used by import-pieces
• writing a per-group completion marker after the aggregate is imported

The aggregate migration is retryable across the important crash points:

• raw bucketing uses a cursor so large inputs do not need to restart from zero
• newly-created raw bucket directory entries are fsynced before cursor commit
• completed JSONL stages are validated by size, record count, and SHA256
• grouping is committed atomically through one groups.jsonl file
• temp aggregate files are treated as uncommitted scratch and can be recreated
• staged aggregate files are trusted only after storacha-aggregate-staging/staged/<piececidv2> exists
• unmarked staged files are treated as uncommitted junk and require manual cleanup
• final aggregate files with matching DB rows are repaired without recomputing CommP
• after the staged marker exists, recovery avoids expensive CommP and trusts marker, DB row, final path, and file size
• corrupt or ambiguous recovery states fail instead of guessing

Adds coverage for aggregate migration behavior, including raw bucket cursor recovery, dedupe, grouping, completed marker resume, temp-file overwrite, staged-file recovery, final-file recovery, missing staged-file cleanup, uncommitted staging cleanup, and final size mismatch cleanup.

How To Use

1. Input file must contain one already-imported Storacha PieceCIDv2 per line:

   <piece-cid-v2>
   <piece-cid-v2>
   ...
   

2. Source must be a Curio-style piece storage directory containing the source pieces:

   <source>/piece/s-t00-<parked_piece_id>

3. Target must be the directory of an attached Curio storage path, for example:

   /path/to/curio-storage

4. Run:

   curio toolbox aggregate-pieces \
     --input /path/to/piececidv2-list.txt \
     --source /path/to/source-storage \
     --target /path/to/curio-storage \
     --result /path/to/aggregate-result.json

   The command writes JSON to --result:

   {
     "aggregates": [
       {
         "piece_cid": "<aggregate-piece-cid-v2>",
         "sub_pieces": ["<subpiece-cid-v2>", "..."],
         "count": 123
       }
     ],
     "error": "optional error string on failed runs"
   }

5. After each invocation, read --result even if the command exits non-zero. The file contains the aggregate pieces successfully completed from the work directory. It may be overwritten by the next run.

6. Re-run until the command exits successfully. If a run returns an error, process any aggregates present in --result first, then retry or stop based on the error.

[TippyFlitsUK]

Heads-up from running this branch in production: the "more parallel IPNI tasks" commit hardcodes Max: taskhelp.Max(50) in PDPv0_IPNI's TypeDetails. That site was since changed by #1293 (merged, in v1.28.2-rc3), and rebasing this PR re-applies the 50 over it — I hit exactly this regression class on 9 June (#1291). Worth dropping that commit or aligning it with the merged state when this rebases. I'm running the branch cherry-picked onto rc3 with the cap pinned low; everything else applied cleanly.

[LexLuthr]

@TippyFlitsUK Let me know when we are good to close this and delete the branch.

[rjan90]

@TippyFlitsUK is still tracking this thread with the migration still ongoing, he will close this issue ticket out once done. I put down a ECD of 202607 in the project tracker.