pdp: serve long-term parked pieces with no deal or pdp ref from the cached reader - #1430
Open
TippyFlitsUK wants to merge 1 commit into
Open
Conversation
…ached reader Migration aggregates are parked long_term with no pdp ref until claimed; the cached reader returned ErrNoDeal for them, 404ing retrievals of complete parked pieces. Fall back to parked_pieces (long_term AND complete) via the piece park reader when no market deal and no pdp ref exists. ErrNoDeal behaviour is unchanged when the fallback finds nothing.
FilOzzy
added
the
team/fs-wg
Collaborator
|
We are basically opening another security hole here. |
Collaborator
|
Following onto what @LexLuthr is saying I think this is not the best fix. I want to first understand what pieces are not being tracked by a pdp_piecref and why. Our first recourse should be to track these pieces correctly. If there is a reason the code is explicitly not tracking them we need to ammend this wholistically. If this is a matter of troubled database state resulting from the understandbly difficult storacha migration then a better option is probably to repair your db. Is there an issue describing the exact problem and if we know why these pieces are not tracked by piecrefs? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Setup: mainnet PDP SPs holding Storacha-migration first copies as long-term parked aggregates (~226k+ pieces), now sourcing second-copy replication pulls. Running v1.28.4.
Retrieving a parked piece that has neither a market deal nor a
pdp_piecerefsrow returnsErrNoDeal, so/piecerequests 404 while the bytes sit complete in the piece park. Migration aggregates are parkedlong_term = TRUEby design with no pdp ref until claimed - stock Curio cannot serve them at all. Hit this on main2.ezpdpz.net on 2026-07-29 (every retrieval of a parked complete aggregate 404'd), and these pieces are exactly what the second-copy repair campaign pulls from a first-copy SP, so on stock the campaign has no source.The fix adds a last-resort fallback in
getPieceReaderFromMarketPieceDeal: when there is no market deal and no pdp ref, look the piece up inparked_pieceswithlong_term = TRUE AND complete = TRUEand serve it via the piece park reader.ErrNoDealis unchanged when the fallback finds nothing.Running in production on both my mainnet SPs since late July, carried locally across v1.28.2-rc3, v1.28.3 and v1.28.4 - currently serving the live second-copy pulls to two other SPs. Happy to adjust or add coverage if wanted.