Checklist
Curio component
Improvement Suggestion
Clients (dealbot, filecoin-pin, FOC upload flows) need to know when it is safe to query cid.contact/cid/{cid} for content uploaded to Curio. Querying too early returns a 404 that seeds IPNI's negative cache, which can delay discoverability by ~20+ minutes even after the content is fully indexed (see filecoin-pin
content-routing FAQ
Curio currently cannot answer "has IPNI actually indexed this ad?":
- GET /pdp/piece/{pieceCid}/status reports retrieved based on ipni_ad_fetches — rows inserted when the indexer fetches the ad from our provider endpoint (market/ipni/ipni-provider/ipni-provider.go, logPDPFetch). A fetch means the indexer pulled the ad, not that it processed/indexed it. It's also unreliable in the best case (fetches can be missed, retried, or served from intermediate state).
- The Publishing head for provider announce log only means we told IPNI a new head exists; ingestion is entirely IPNI-driven after that.
- this means tracking and benchmarking index latency also very challenging;
storetheindex now exposes (ipni/storetheindex#2859 PR #2861 deployed on cid.contact:
$ curl -s https://cid.contact/sync/status/ad/baguqeera... | jq .
{
"Ad": "baguqeera...",
"Indexed": true
}
Indexed: true means the indexer has fully processed that advertisement. With this, Curio can gate its "IPNI-ready" signal on confirmed ingestion, and clients polling the piece status endpoint never need to touch /cid/{cid} before it's positive;
Proposal (to be reviewed and corrected by @LexLuthr )
- a lightweight periodic task that, for each provider (peer ID) with unconfirmed announced ads, queries {ServiceURL}/sync/status/ad/{adCid} and records confirmation.
2.l record confirmation to allow us tracking ad and piece index status and latency;
- Surface in GET /pdp/piece/{pieceCid}/status (pdp/handlers.go, handleGetPieceStatus): add synced / syncedAt fields and a new terminal status: pending → indexing → creating_ad → announced → synced
- Metrics/logging: log ad-confirmed events with timestamps so we can measure announce→indexed latency end to end (announce time is already logged with
log.Infow("Publishing head for provider", "provider", provider, "cid", c.String()) ).
Others
- I would love to have some benchmarking/metric from a curio ipni publisher that what is the
time to availability is like, meaning how long does a cid can be looked up from ipni since we have published the ad;
- this will help us evaluate if IPNI is performant enough for us to drop customized pin.contact
Checklist
Latest release, or the most recent RC(release candidate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.Curio component
Improvement Suggestion
Clients (dealbot, filecoin-pin, FOC upload flows) need to know when it is safe to query cid.contact/cid/{cid} for content uploaded to Curio. Querying too early returns a 404 that seeds IPNI's negative cache, which can delay discoverability by ~20+ minutes even after the content is fully indexed (see filecoin-pin
content-routing FAQ
Curio currently cannot answer "has IPNI actually indexed this ad?":
storetheindex now exposes (ipni/storetheindex#2859 PR #2861 deployed on cid.contact:
Indexed: true means the indexer has fully processed that advertisement. With this, Curio can gate its "IPNI-ready" signal on confirmed ingestion, and clients polling the piece status endpoint never need to touch /cid/{cid} before it's positive;
Proposal (to be reviewed and corrected by @LexLuthr )
2.l record confirmation to allow us tracking ad and piece index status and latency;
log.Infow("Publishing head for provider", "provider", provider, "cid", c.String())).Others
time to availabilityis like, meaning how long does a cid can be looked up from ipni since we have published the ad;