Skip to content

pdpv0: Document PDP task inventory and dependency tree #998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ZenGround0 merged 3 commits into from
Feb 18, 2026
Merged

pdpv0: Document PDP task inventory and dependency tree #998

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c36c644
pdpv0: document PDP task inventory and dependency tree
rjan90 Feb 16, 2026
5226926
pdpv0: address PR review feedback on PDP spec
rjan90 Feb 18, 2026
1a35d00
Review touch up
ZenGround0 Feb 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
159 changes: 159 additions & 0 deletions documentation/en/experimental-features/PDPCURIOSPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,162 @@ TODO: focus on transitions between this and next proving period not the proving
## TODO: more categories

# Payment Settlement

# PDP Harmony Tasks

All PDP-related functionality is implemented as harmony tasks and chain-handler watchers. Tasks implement the `TaskInterface` and are scheduled by the harmony task engine. Chain-handler watchers are registered via `chainsched.AddHandler` and fire on every new chain head.
Copy link
Collaborator

@ZenGround0 ZenGround0 Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love seeing the chainsched handlers mentioned. Since we are getting into the mechanis I think we should probably have more exposition on how exactly harmony tasks are scheduled by the task engine -- i.e. IAMBored or constructors creating chain handlers that spawn harmony tasks.

Copy link
Collaborator

@ZenGround0 ZenGround0 Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok nice I see that this is called out in the registered task table's Trigger column. But some exposition on the interaction between chain handler and tasks and the broader points of poller vs IAMBored from harmony task would be useful here


## Registered Tasks

| Task Name | Struct | File | Trigger | Description |
|-----------|--------|------|---------|-------------|
| `PDPv0_InitPP` | `InitProvingPeriodTask` | `tasks/pdp/task_init_pp.go` | Chain handler | Initializes the first proving period for a dataset by calling `PDPVerifier.nextProvingPeriod()`. Picks up datasets where `init_ready=TRUE` and `prove_at_epoch IS NULL`. |
| `PDPv0_ProvPeriod` | `NextProvingPeriodTask` | `tasks/pdp/task_next_pp.go` | Chain handler | Schedules subsequent proving periods after each challenge window completes. Fires when `prove_at_epoch + challenge_window <= current_height`. Detects `ProvingPeriodNotInitialized` errors and resets dataset back to init state. |
| `PDPv0_Prove` | `ProveTask` | `tasks/pdp/task_prove.go` | Chain handler | Generates SHA-256 Merkle tree proofs and submits them via `PDPVerifier.provePossession()`. Scheduled when the `challenge_request_msg_hash` transaction lands on-chain and `prove_at_epoch` is reached. Most resource-intensive task (2GB RAM for memtree). |
Copy link
Collaborator

@ZenGround0 ZenGround0 Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my POV we could just drop the description field. The descriptions there are ok but mixed between very high level and fine details which makes reading them a bit tricky at least for me. My thinking would be that the rest of this document should be describing these tasks in enough detail that anyone reading this will already understand what they are doing. A reference table listing trigger type is quite useful though.

| `PDPv0_Notify` | `PDPNotifyTask` | `tasks/pdp/notify_task.go` | Polling (2s) | Finalizes completed piece uploads by sending an optional HTTP callback to `notify_url` and moving the piece reference from temporary `pdp_piece_uploads` to permanent `pdp_piecerefs`. |
| `PDPv0_PullPiece` | `PDPPullPieceTask` | `tasks/pdp/task_pull_piece.go` | Polling (10s) | Downloads pieces from external URLs, streams data while computing CommP, verifies against expected piece CID, and stores in `parked_pieces` for long-term storage. |
| `PDPv0_TermFWSS` | `TerminateFWSSTask` | `tasks/pdp/task_terminate_fwss.go` | IAmBored (1 min) | Calls `FWSS.terminateService()` for datasets pending service termination. Retrieves `PdpEndEpoch` from the FWSS contract. |
| `PDPv0_DelDataSet` | `DeleteDataSetTask` | `tasks/pdp/task_delete_data_set.go` | IAmBored (1 hour) | Calls `PDPVerifier.deleteDataSet()` for datasets where `service_termination_epoch <= current_block` and `deletion_allowed=TRUE`. |
| `Settle` | `SettleTask` | `tasks/pay/settle_task.go` | IAmBored (12 hours) | Settles the FWSS lockup period for the PDP operator. Calls `filecoinpayment.SettleLockupPeriod()` with the operator and payee addresses. |
Copy link
Collaborator

@ZenGround0 ZenGround0 Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add PDPv0_Indexing and PDPv0_IPNI


## Chain-Handler Watchers

These are not harmony tasks but chain-event handlers registered via `chainsched.AddHandler`. They fire on every new chain head and process pending on-chain transactions.

| Watcher | File | Description |
|---------|------|-------------|
| `DataSetWatch` | `tasks/pdp/dataset_watch.go` | Runs `processPendingDataSetCreates` then `processPendingDataSetPieceAdds` sequentially. Extracts `DataSetCreated` and `PiecesAdded` events from transaction receipts. Sets `init_ready=TRUE` on first piece addition. |
| `TerminateServiceWatcher` | `tasks/pdp/watch_fwss_terminate.go` | Monitors `terminateService` transaction completion. Retrieves `PdpEndEpoch` from the FWSS contract and updates `service_termination_epoch` in `pdp_delete_data_set`. |
| `DataSetDeleteWatcher` | `tasks/pdp/watch_data_set_delete.go` | Monitors `deleteDataSet` transaction completion. Verifies dataset is no longer live on-chain, then deletes the `pdp_data_sets` row (CASCADE deletes pieces and prove tasks). |
| `PieceDeleteWatcher` | `tasks/pdp/watch_piece_delete.go` | Monitors piece removal transaction completion. Marks pieces as `removed=TRUE`, cleans up `pdp_piecerefs` after a 24-hour grace period, and publishes IPNI removal advertisements. |
| `SettleWatcher` | `tasks/pay/watcher.go` | Monitors settlement transaction completion. Verifies settlement status on-chain via the FWSS contract. Detects terminated or defaulting rails and triggers service termination and dataset deletion via `pdp.EnsureServiceTermination` and `ensureDataSetDeletion`. |

# Task Dependency Tree

The following diagrams show how tasks and watchers connect to form the PDP lifecycle. Arrows represent causal dependencies: the source produces a database state or on-chain event that the target consumes.

## Piece Ingestion

```
HTTP Upload HTTP Pull Request
│ │
▼ ▼
parked_pieces pdp_piece_pull_items
(upload in progress) │
│ ▼
│ PDPv0_PullPiece (10s poll)
│ ── downloads piece
│ ── verifies CommP
│ │
│ ▼
│ parked_pieces + pdp_piecerefs
│
▼
parked_pieces.complete=TRUE
│
▼
PDPv0_Notify (2s poll)
── sends callback to notify_url
── moves pdp_piece_uploads → pdp_piecerefs
```

## Dataset Creation and Proving Cycle

```
HTTP: createDataSet()
│
▼
pdp_data_set_creates (create_message_hash)
│
▼ on-chain tx lands
DataSetWatch (chain handler)
── processPendingDataSetCreates: extracts dataSetId from receipt
── processPendingDataSetPieceAdds: extracts pieceIds, sets init_ready=TRUE
│
▼ init_ready=TRUE, prove_at_epoch IS NULL
PDPv0_InitPP (chain handler)
── calls PDPVerifier.nextProvingPeriod()
── sets prove_at_epoch, challenge_request_msg_hash
│
▼ challenge_request_msg_hash tx lands, prove_at_epoch reached
PDPv0_Prove (chain handler)
── builds Merkle tree proofs
── calls PDPVerifier.provePossession()
── resets challenge_request_msg_hash to NULL
│
▼ prove_at_epoch + challenge_window <= current_height
PDPv0_ProvPeriod (chain handler)
── calls PDPVerifier.nextProvingPeriod()
── sets new prove_at_epoch, challenge_request_msg_hash
│
└──────► loops back to PDPv0_Prove
```

## Payment Settlement and Termination

```
Settle (every 12 hours)
── calls filecoinpayment.SettleLockupPeriod()
── writes tx to filecoin_payment_transactions + message_waits_eth
│
▼ tx lands
SettleWatcher (chain handler)
── verifies settlement on-chain
── detects terminated/defaulting rails
│
├──► pdp.EnsureServiceTermination() (rail terminated or near default)
│ │
│ ▼ pdp_delete_data_set row created
│ PDPv0_TermFWSS (every 1 min)
│ ── calls FWSS.terminateService()
│ │
│ ▼ tx lands
│ TerminateServiceWatcher (chain handler)
│ ── retrieves PdpEndEpoch from FWSS
│ ── sets service_termination_epoch
│ │
│ ▼ service_termination_epoch <= current_block, deletion_allowed=TRUE
│ PDPv0_DelDataSet (every 1 hour)
│ ── calls PDPVerifier.deleteDataSet()
│ │
│ ▼ tx lands
│ DataSetDeleteWatcher (chain handler)
│ ── verifies dataset no longer live
│ ── deletes pdp_data_sets (CASCADE)
│
└──► ensureDataSetDeletion() (rail fully settled/finalized)
sets deletion_allowed=TRUE in pdp_delete_data_set
picked up by PDPv0_DelDataSet above
```

## Piece Removal

```
HTTP: removePieces()
── sends PDPVerifier.removePieces() tx
│
▼ tx lands
PieceDeleteWatcher (chain handler)
── marks pdp_data_set_pieces.removed=TRUE
── after 24h grace: cleans up pdp_piecerefs, parked_piece_refs
── publishes IPNI removal advertisements
```

## Error Recovery

```
PDPv0_Prove failure
│
├── Transient error ──► harmony retry (up to 5 attempts)
│
├── Contract revert ──► exponential backoff
│ (100 blocks * 2^(failures-1), max 28800 blocks)
│ after 5 consecutive failures ──► unrecoverable
│
Copy link
Collaborator

@ZenGround0 ZenGround0 Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps just my opinion but I get the sense that these ASCII diagrams are too crowded to be very useful. I think a bulleted list or text description would probably better suite presenting this information. Alternatively a more simplified ascii diagram with only arrows between tasks could also be a good direction to take this.

└── Unrecoverable error ──► sets unrecoverable_proving_failure_epoch
dataset stops proving

PDPv0_ProvPeriod detects ProvingPeriodNotInitialized
└──► resets dataset to init state (init_ready=TRUE, prove_at_epoch=NULL)
picked up by PDPv0_InitPP
```