Skip to content

Use boxo with provider queue input buffer#10758

Closed
gammazero wants to merge 8 commits intomasterfrom
faster-online-add
Closed

Use boxo with provider queue input buffer#10758
gammazero wants to merge 8 commits intomasterfrom
faster-online-add

Conversation

@gammazero
Copy link
Copy Markdown
Contributor

No description provided.

@gammazero gammazero added the skip/changelog This change does NOT require a changelog entry label Mar 19, 2025
@lidel
Copy link
Copy Markdown
Member

lidel commented Mar 20, 2025

Notes form colo:

  • right now, by default provider queue is being persisted in leveldb datastore, which introduces extra disk writes and reads, as noted in Improve performance of data onboarding boxo#888 (comment)
  • we want Kubo to have an option to avoid doing disk writes when provide system works
    • (idea 1) we could ensure provider queue is persisted under deterministic queue
      • this would allow user to control where provide queue is stored, for example, user could mount in-memory datastore under something like /provides – similar to how flatfs is mounted for `/blocks
    • (idea 2) Reprovider.QueuePersistence Flag that allows user to control if queue is persisted in datastore, or not (prioritizing performance at the cost or slight risk in case of power outage and increased memory usage)

@gammazero gammazero force-pushed the faster-online-add branch from 353f93c to b11c382 Compare April 2, 2025 23:11
@gammazero gammazero closed this Apr 6, 2025
@gammazero gammazero deleted the faster-online-add branch April 6, 2025 06:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip/changelog This change does NOT require a changelog entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants