Skip to content

Integrate IPNI /sync/status/ad/{adCid} to track confirmed advertisement indexing #1405

Description

@jennijuju

Checklist

  • This is not a question or a support request. If you have any Curio related questions, please ask in the discussion forum.
  • This is not a new feature request. If it is, please file a feature request instead.
  • I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
  • I am running the Latest release, or the most recent RC(release candidate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.

Curio component

  • PoRep Pipeline
  • Snap Deals
  • Market
  • IPNI provider
  • Curio UI
  • Other

Improvement Suggestion

Clients (dealbot, filecoin-pin, FOC upload flows) need to know when it is safe to query cid.contact/cid/{cid} for content uploaded to Curio. Querying too early returns a 404 that seeds IPNI's negative cache, which can delay discoverability by ~20+ minutes even after the content is fully indexed (see filecoin-pin
content-routing FAQ

Curio currently cannot answer "has IPNI actually indexed this ad?":

  • GET /pdp/piece/{pieceCid}/status reports retrieved based on ipni_ad_fetches — rows inserted when the indexer fetches the ad from our provider endpoint (market/ipni/ipni-provider/ipni-provider.go, logPDPFetch). A fetch means the indexer pulled the ad, not that it processed/indexed it. It's also unreliable in the best case (fetches can be missed, retried, or served from intermediate state).
  • The Publishing head for provider announce log only means we told IPNI a new head exists; ingestion is entirely IPNI-driven after that.
  • this means tracking and benchmarking index latency also very challenging;

storetheindex now exposes (ipni/storetheindex#2859 PR #2861 deployed on cid.contact:

  $ curl -s https://cid.contact/sync/status/ad/baguqeera... | jq .
  {
    "Ad": "baguqeera...",
    "Indexed": true
  }

Indexed: true means the indexer has fully processed that advertisement. With this, Curio can gate its "IPNI-ready" signal on confirmed ingestion, and clients polling the piece status endpoint never need to touch /cid/{cid} before it's positive;

Proposal (to be reviewed and corrected by @LexLuthr )

  1. a lightweight periodic task that, for each provider (peer ID) with unconfirmed announced ads, queries {ServiceURL}/sync/status/ad/{adCid} and records confirmation.
    2.l record confirmation to allow us tracking ad and piece index status and latency;
  2. Surface in GET /pdp/piece/{pieceCid}/status (pdp/handlers.go, handleGetPieceStatus): add synced / syncedAt fields and a new terminal status: pending → indexing → creating_ad → announced → synced
  3. Metrics/logging: log ad-confirmed events with timestamps so we can measure announce→indexed latency end to end (announce time is already logged with log.Infow("Publishing head for provider", "provider", provider, "cid", c.String()) ).

Others

  • I would love to have some benchmarking/metric from a curio ipni publisher that what is the time to availability is like, meaning how long does a cid can be looked up from ipni since we have published the ad;
  • this will help us evaluate if IPNI is performant enough for us to drop customized pin.contact

Metadata

Metadata

Assignees

Labels

team/fs-wgItems being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10

Type

No type

Projects

Status
⌨ In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions