repair cli

[hugomrdias]

• add filter for dataset with pdpEndEpoch > current block (save block number to repair table)
• commit pieces
• validate retry on sp rate limits 429
• move serviceUrl out of operations and into repair
• enable WAL for sqlite PRAGMA journal_mode = WAL
• ignore non endorsed and non approved SPs ?
• add progress output
• graceful exit

closes #6

[Kubuxu]

I've run the repair process, it failed to create datasets due to insufficient funds, and then, after increasing funds in FilecoinPay, it continued to fail because datasets did not exist. It didn't re-try creating the dataset.

[Kubuxu]

I tried running repair from ID=1 to ID=2 on calibnet (ID=1 is my old node):
getting:

Failed to pull pieces from storage provider.

Details: Service Provider PDP
Invalid extraData payer: decode combined extraData: abi: cannot marshal in to go slice: offset 49935820931146658734213847930527602599075759025152894063139027855094479731196 would go over slice boundary (len=1056) [eid: b0de651f-09ea-42b8-a91b-0ecb6fc5d38c]

Failed to pull pieces from storage provider.

Details: Service Provider PDP
Invalid extraData payer: decode combined extraData: abi: cannot marshal in to go slice: offset 61933373589282278564299983318175307680785193666285889285970360447521741517436 would go over slice boundary (len=448) [eid: 1f8d5ba8-1d56-429f-a52a-615dcac397eb]

Failed to pull pieces from storage provider.

Details: Service Provider PDP
Invalid extraData payer: decode combined extraData: abi: cannot marshal in to go slice: offset 92099208621840918449134908834052895815613600926412686856706603646053003229750 would go over slice boundary (len=1824) [eid: d7bfbc43-4fcf-4145-b310-9d7ef69fec46]

sqlite> select * from operations where repair_id = 4 and status != 'skipped';
48084|4|add_piece|pending|cdn|{"cid":"bafkzcibfqsmoaeywv6zwbneppp43abhozqxeeragpswrfs6g3cudytdgyqxit3c7gqaa","metadata":{"name":"Screen Recording 2024-07-29 at 1.02.15 PM.mov","type":"video/quicktime"},"alternateProviders":["https://calib2.ezpdpz.net","https://caliberation-pdp.infrafolio.com","https://calib.ezpdpz.net"]}|||1779799701627|1779799701627
48086|4|add_piece|pending|cdn|{"cid":"bafkzcibdvvrqwb6olvqtb5tuuo5gvqprcabo33k5uq7lvev5xsog22iqwh5u42bs","metadata":{"name":"piece-bafkzcibdvvrqwb6olvqtb5tuuo5gvqprcabo33k5uq7lvev5xsog22iqwh5u42bs (2).jpeg","type":"image/jpeg"},"alternateProviders":["https://calib.ezpdpz.net"]}|||1779799701627|1779799701627
48088|4|add_piece|pending|cdn|{"cid":"bafkzcibevdmd4ew4oyakxbdiirchrm3djug4pa46iopdnxgtoeuwfmnmwt2dvmdveq","metadata":{},"alternateProviders":["https://calib2.ezpdpz.net","https://calib.ezpdpz.net"]}|||1779799701627|1779799701627
48090|4|add_piece|pending|cdn|{"cid":"bafkzcibfrkfoahqweihjbacf5tvdli5z5o6klsq3mpq2vdcrohtx4jptyncluae7cqmq","metadata":{},"alternateProviders":["https://caliberation-pdp.infrafolio.com","https://calib.ezpdpz.net","https://calibnet.pspsps.io","https://pdp.topblocks.io","https://rukapdp.drongyl.com","https://pdp.laughstorage.com"]}|||1779799701627|1779799701627
48095|4|add_piece|pending|none|{"cid":"bafkzcibdxexasatdtgin6xmoh52qk7ahdgz35amzfc72vjjddkej2dec3s726mbp","metadata":{},"alternateProviders":["https://calib2.ezpdpz.net","https://caliberation-pdp.infrafolio.com","https://pdp-calib.filweb3.com","https://calibnet.pspsps.io","https://pdp.oplian.com","https://pdp.topblocks.io","https://rukapdp.drongyl.com","https://pdp.laughstorage.com"]}|||1779799701627|1779799701627
48097|4|add_piece|pending|cdn|{"cid":"bafkzcibdsabqq2an2kba2n2k5dk6abs6fhayd7otmyppqeopwj5q3wnkglualtyb","metadata":{},"alternateProviders":["https://calib2.ezpdpz.net","https://calib.ezpdpz.net"]}|||1779799701627|1779799701627
48145|4|add_piece|pending|ipfs|{"cid":"bafkzcibevxragdmi5eqwxf4pdw6syjirkzkclseiddtd6ljaf65d3fnbmynmqov3eu","metadata":{"label":"image (2).png","fileSize":"193.51 KB","ipfsRootCID":"bafybeih7qx6wfldylvvqirl6w6bgczm7n4sjwfnkkqtktvwp23rg27pa5y"},"alternateProviders":["https://warp.lotus.dedyn.io"]}|||1779799701627|1779799701627
48160|4|create_dataset|completed|cdn|{"payee":"0xbcdf1bdc1a97d071a5a8ef03f1f05225b6e2a1ba"}|{"txHash":"0x9a34aab4ec7c82c04f68ea722350e7e0041cb2c42f19aed6bd68a04e2ba8894c","dataSetId":{"$bigint":"14013"}}||1779799701280|1779799778201
48161|4|create_dataset|completed|ipfs|{"payee":"0xbcdf1bdc1a97d071a5a8ef03f1f05225b6e2a1ba"}|{"txHash":"0x822b1b9e20451ff46a69f51a4d4e87634bce0e7b9b1d30c8afab0cb51c26e22d","dataSetId":{"$bigint":"14014"}}||1779799701280|1779799778449
48162|4|create_dataset|completed|none|{"payee":"0xbcdf1bdc1a97d071a5a8ef03f1f05225b6e2a1ba"}|{"txHash":"0x514e4ed24dadbbcca2dec252035164e3786a8e51ce1cd1eb7afc08d22c05e8bc","dataSetId":{"$bigint":"14012"}}||1779799701280|1779799778347

[BigLep]

@silent-cipher: I'd like to have you look at this PR. You'll need to accept the invite to the repo so we can assign you as a reviewer.

[Copilot]

Pull request overview

Adds a new @filoz/repair-cli package implementing an “early repair” workflow that plans and runs repairs by reading provider/dataset/piece state from a Postgres indexer DB and tracking execution state in a local SQLite DB.

Changes:

• Introduces repair CLI commands (setup, wallet, repair, datasets, providers) plus middleware/context wiring.
• Adds local SQLite + indexer Postgres Drizzle schemas and a set of DB helpers to plan repairs and page through pieces/providers.
• Implements initial pipeline phases for dataset creation and piece pulling, plus supporting utilities (config, migrations, JSON helpers, upsert helpers).

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
pnpm-workspace.yaml	Updates workspace catalog versions and allows native build for better-sqlite3.
packages/repair-cli/tsconfig.json	Includes package.json for TS JSON import support.
packages/repair-cli/src/utils.ts	Adds CLI config, wallet client creation, migration helper, metadata helpers, JSON (de)serialization, and misc utilities.
packages/repair-cli/src/types.ts	Defines shared types for context, DB handles, piece groups, and provider shape.
packages/repair-cli/src/pipeline/pull.ts	Adds pull phase batching/queueing and a pull worker implementation.
packages/repair-cli/src/pipeline/create-datasets.ts	Adds dataset creation phase worker/queue and repair dataset validation.
packages/repair-cli/src/middleware.ts	Adds context middleware to wire local/indexer DBs + wallet client into commands.
packages/repair-cli/src/local-schema.ts	Introduces SQLite schema for repairs/operations with custom JSON/bigint types.
packages/repair-cli/src/indexer-schema.ts	Introduces Postgres schema for providers/datasets/pieces in early-repair.
packages/repair-cli/src/error.ts	Adds typed error classes for repair creation/runtime failures.
packages/repair-cli/src/db/upsert-operations.ts	Adds helper to upsert operation status updates.
packages/repair-cli/src/db/update-repair.ts	Adds helper to update repair status/target datasets.
packages/repair-cli/src/db/update-operation.ts	Adds helper to update operation status/result/error.
packages/repair-cli/src/db/select-alternate-repair-provider.ts	Adds automatic selection of an alternate repair target provider with tier fallback.
packages/repair-cli/src/db/get-target-dataset.ts	Adds helper to load/cache target dataset by repair+group.
packages/repair-cli/src/db/get-repair-provider.ts	Adds helper to load an explicit target provider by ID.
packages/repair-cli/src/db/get-repair-groups.ts	Adds helper to derive repair groups from pending local operations.
packages/repair-cli/src/db/get-providers-by-cid.ts	Adds helper to find alternate providers per CID via indexer joins.
packages/repair-cli/src/db/get-pieces.ts	Adds paginated piece walk and mapping to local add_piece operations.
packages/repair-cli/src/db/get-datasets-by-group.ts	Adds helper to find one dataset per group for payer+source.
packages/repair-cli/src/db/get-dataset-for-group.ts	Adds helper to find a dataset for a single group.
packages/repair-cli/src/db/filter-pull-pieces-not-in-dataset.ts	Adds helper to filter out CIDs already present in a dataset.
packages/repair-cli/src/db/delete-repair.ts	Adds helper to delete a repair and its operations.
packages/repair-cli/src/db/create-repair.ts	Adds orchestration to plan a repair (target selection, operations creation, dataset ops).
packages/repair-cli/src/commands/wallet.ts	Adds wallet-related commands (fund, balance, deposit, withdraw).
packages/repair-cli/src/commands/setup.ts	Adds interactive setup and local DB migration.
packages/repair-cli/src/commands/repair.ts	Adds repair commands (create/list/delete/run).
packages/repair-cli/src/commands/providers.ts	Adds provider listing command (active providers).
packages/repair-cli/src/commands/datasets.ts	Adds dataset listing command with optional provider filter.
packages/repair-cli/src/cli.ts	Adds CLI entrypoint and command registration.
packages/repair-cli/package.json	Adds dependencies and typing deps for the new CLI package.
packages/repair-cli/AGENTS.md	Adds agent/operator documentation for the repair CLI and pipeline.
package.json	Removes an unused script and pnpm overrides block.
apps/ponder/package.json	Pins drizzle-orm version for the ponder app.
.vscode/settings.json	Sets workspace TypeScript SDK path for consistent editor TS version.
.agents/skills/incur/SKILL.md	Adds incur framework skill documentation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.