Skip to content

feat: Provider.WorkerCount and stats reprovide cmd #10779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lidel merged 16 commits into from
Apr 30, 2025
Merged

feat: Provider.WorkerCount and stats reprovide cmd #10779

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
05d0bb3
adjust ipfs stats provide
guillaumemichel Apr 9, 2025
4e8d223
update boxo dep
guillaumemichel Apr 16, 2025
0e37989
Merge branch 'master' into adjust-stats-provide
guillaumemichel Apr 16, 2025
62d51d6
update boxo dep
guillaumemichel Apr 16, 2025
b33c4fa
provider worker count config
guillaumemichel Apr 16, 2025
b0ba27c
adjust config logic
guillaumemichel Apr 16, 2025
0662522
bump boxo
guillaumemichel Apr 24, 2025
cc02022
docs
guillaumemichel Apr 24, 2025
5d148b8
address review
guillaumemichel Apr 29, 2025
617d5c8
fixing tests
guillaumemichel Apr 29, 2025
c65608c
adapt sharness tests
guillaumemichel Apr 29, 2025
30bdc15
fix(config): skip unset Provide.Strategy
lidel Apr 30, 2025
9a04312
docs/chore: mark stat reprovide as experimental
lidel Apr 30, 2025
06e6618
docs: Provider.Strategy
lidel Apr 30, 2025
b0f3bbd
Merge 'origin/master' into adjust-stats-provide
lidel Apr 30, 2025
74cf6af
chore: fix typo
lidel Apr 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion config/provider.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
package config

const (
DefaultProviderWorkerCount = 64
)

type Provider struct {
Strategy string // Which keys to announce
Strategy string // Which keys to announce
Comment thread
guillaumemichel marked this conversation as resolved.
Outdated
WorkerCount OptionalInteger // Number of concurrent provides allowed, 0 means unlimited
Comment thread
guillaumemichel marked this conversation as resolved.
Outdated
}
8 changes: 4 additions & 4 deletions core/commands/stat_provide.go
View file
Open in desktop
Copy link
Copy Markdown
Member

@lidel lidel Apr 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: renaming field names here is a breaking change to existing command.

It will break compatibility with older RPC clients, libraries, and will also break user's scripts and automations.

Mind adding a note about this to the Changelog?

Since we have two queues, and they behave differently, I wonder if we should have ipfs stat provide and ipfs stat reprovide? (e.g. if we make breaking change anyway, any reason to not rename current provide to reprovide?)

Copy link
Copy Markdown
Contributor Author

@guillaumemichel guillaumemichel Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I marked ipfs stats provide as deprecated, and renamed it to ipfs stats reprovide.

ipfs stats provide shows the same stats as ipfs stats reprovide but with the old names (only information related to reprovides, because we don't track provides anymore), so it shouldn't break existing scripts.

Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,13 @@ This interface is not stable and may change from release to release.
wtr := tabwriter.NewWriter(w, 1, 2, 1, ' ', 0)
defer wtr.Flush()

fmt.Fprintf(wtr, "TotalProvides:\t%s\n", humanNumber(s.TotalProvides))
fmt.Fprintf(wtr, "AvgProvideDuration:\t%s\n", humanDuration(s.AvgProvideDuration))
fmt.Fprintf(wtr, "TotalReprovides:\t%s\n", humanNumber(s.TotalReprovides))
fmt.Fprintf(wtr, "AvgReprovideDuration:\t%s\n", humanDuration(s.AvgReprovideDuration))
fmt.Fprintf(wtr, "LastReprovideDuration:\t%s\n", humanDuration(s.LastReprovideDuration))
if !s.LastRun.IsZero() {
fmt.Fprintf(wtr, "LastRun:\t%s\n", humanTime(s.LastRun))
fmt.Fprintf(wtr, "LastReprovide:\t%s\n", humanTime(s.LastRun))
if s.fullRT {
fmt.Fprintf(wtr, "NextRun:\t%s\n", humanTime(s.LastRun.Add(s.ReprovideInterval)))
fmt.Fprintf(wtr, "NextReprovide:\t%s\n", humanTime(s.LastRun.Add(s.ReprovideInterval)))
}
}
return nil
Expand Down
1 change: 1 addition & 0 deletions core/node/groups.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,7 @@ func Online(bcfg *BuildCfg, cfg *config.Config, userResourceOverrides rcmgr.Part
cfg.Reprovider.Strategy.WithDefault(config.DefaultReproviderStrategy),
cfg.Reprovider.Interval.WithDefault(config.DefaultReproviderInterval),
cfg.Routing.AcceleratedDHTClient.WithDefault(config.DefaultAcceleratedDHTClient),
int(cfg.Provider.WorkerCount.WithDefault(config.DefaultProviderWorkerCount)),
),
)
}
Expand Down
8 changes: 4 additions & 4 deletions core/node/provider.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,13 @@ import (
// and in 'ipfs stats provide' report.
const sampledBatchSize = 1000

func ProviderSys(reprovideInterval time.Duration, acceleratedDHTClient bool) fx.Option {
func ProviderSys(reprovideInterval time.Duration, acceleratedDHTClient bool, provideWorkerCount int) fx.Option {
return fx.Provide(func(lc fx.Lifecycle, cr irouting.ProvideManyRouter, keyProvider provider.KeyChanFunc, repo repo.Repo, bs blockstore.Blockstore) (provider.System, error) {
opts := []provider.Option{
provider.Online(cr),
provider.ReproviderInterval(reprovideInterval),
provider.KeyProvider(keyProvider),
provider.ProvideWorkerCount(provideWorkerCount),
}
if !acceleratedDHTClient && reprovideInterval > 0 {
// The estimation kinda suck if you are running with accelerated DHT client,
Expand Down Expand Up @@ -131,7 +132,7 @@ https://github.com/ipfs/kubo/blob/master/docs/config.md#routingaccelerateddhtcli
// ONLINE/OFFLINE

// OnlineProviders groups units managing provider routing records online
func OnlineProviders(useStrategicProviding bool, reprovideStrategy string, reprovideInterval time.Duration, acceleratedDHTClient bool) fx.Option {
func OnlineProviders(useStrategicProviding bool, reprovideStrategy string, reprovideInterval time.Duration, acceleratedDHTClient bool, provideWorkerCount int) fx.Option {
if useStrategicProviding {
return OfflineProviders()
}
Expand All @@ -146,7 +147,7 @@ func OnlineProviders(useStrategicProviding bool, reprovideStrategy string, repro

return fx.Options(
keyProvider,
ProviderSys(reprovideInterval, acceleratedDHTClient),
ProviderSys(reprovideInterval, acceleratedDHTClient, provideWorkerCount),
)
}

Expand All @@ -169,7 +170,6 @@ func mfsProvider(mfsRoot *mfs.Root, fetcher fetcher.Factory) provider.KeyChanFun
kcf := provider.NewDAGProvider(rootNode.Cid(), fetcher)
return kcf(ctx)
}

}

func mfsRootProvider(mfsRoot *mfs.Root) provider.KeyChanFunc {
Expand Down
20 changes: 20 additions & 0 deletions docs/changelogs/v0.35.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ This release was brought to you by the [Shipyard](http://ipshipyard.com/) team.
- [New `ipfs add` Options](#new-ipfs-add-options)
- [Persistent `Import.*` Configuration](#persistent-import-configuration)
- [Updated Configuration Profiles](#updated-configuration-profiles)
- [Provide Regression Fix](#provide-regression-fix)
- [📦️ Important dependency updates](#-important-dependency-updates)
- [📝 Changelog](#-changelog)
- [👨‍👩‍👧‍👦 Contributors](#-contributors)
Expand Down Expand Up @@ -81,6 +82,25 @@ The release updated configuration [profiles](https://github.com/ipfs/kubo/blob/m
> [!TIP]
> Apply one of CIDv1 test [profiles](https://github.com/ipfs/kubo/blob/master/docs/config.md#profiles) with `ipfs config profile apply test-cid-v1[-wide]`.

#### Provide Regression Fix
Comment thread
guillaumemichel marked this conversation as resolved.
Outdated

From `kubo` [`v0.33.0`](https://github.com/ipfs/kubo/releases/tag/v0.33.0),
Bitswap stopped advertising newly added and received blocks to the DHT. Since
then `boxo/provider` is responsible for the provide and reprovide logic. Prior
to `v0.35.0`, provides and reprovides were handled together in batches, leading
to delays in initial advertisements (provides).

Provides and Reprovides now have separate queues, allowing for immediate
provide of new CIDs and optimised batching of reprovides.

This change introduces a new configuration option for limiting the number of
concurrent provide operations:
[`Provider.WorkerCount`](https://github.com/ipfs/kubo/blob/master/docs/config.md#providerworkercount).

> [!TIP]
> Users who need to provide large volumes of content immediately should
> consider removing the cap on concurrent provide operations.

#### 📦️ Important dependency updates

- update `ipfs-webui` to [v4.7.0](https://github.com/ipfs/ipfs-webui/releases/tag/v4.7.0)
Expand Down
25 changes: 25 additions & 0 deletions docs/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ config file at runtime.
- [`Pinning.RemoteServices: Policies.MFS.Enabled`](#pinningremoteservices-policiesmfsenabled)
- [`Pinning.RemoteServices: Policies.MFS.PinName`](#pinningremoteservices-policiesmfspinname)
- [`Pinning.RemoteServices: Policies.MFS.RepinInterval`](#pinningremoteservices-policiesmfsrepininterval)
- [`Provider`](#provider)
- [`Provider.WorkerCount`](#providerworkercount)
- [`Pubsub`](#pubsub)
- [`Pubsub.Enabled`](#pubsubenabled)
- [`Pubsub.Router`](#pubsubrouter)
Expand Down Expand Up @@ -1404,6 +1406,29 @@ Default: `"5m"`

Type: `duration`

## `Provider`

Comment thread
guillaumemichel marked this conversation as resolved.
### `Provider.WorkerCount`

Sets the maximum number of _concurrent_ DHT provide operations. DHT reprovides
operations do **not** count against that limit. A value of `0` allows an
unlimited number of provide workers.

If the [accelerated DHT client](#routingaccelerateddhtclient) is enabled, each
provide operation opens ~20 connections in parallel. With the standard DHT
client (accelerated disabled), each provide opens between 20 and 60
connections, with at most 10 active at once. Provides complete more quickly
when using the accelerated client. Be mindful of how many simultaneous
connections this setting can generate.

For nodes without strict connection limits that need to provide large volumes
of content immediately, we recommend enabling the accelerated DHT client and
setting `Provider.WorkerCount` to `0` (unlimited).

Default: `64`

Type: `integer` (non-negative; `0` means unlimited number of workers)

## `Pubsub`

**DEPRECATED**: See [#9717](https://github.com/ipfs/kubo/issues/9717)
Expand Down
2 changes: 1 addition & 1 deletion docs/examples/kubo-as-a-library/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ go 1.24
replace github.com/ipfs/kubo => ./../../..

require (
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161
github.com/ipfs/kubo v0.0.0-00010101000000-000000000000
github.com/libp2p/go-libp2p v0.41.1
github.com/multiformats/go-multiaddr v0.15.0
Expand Down
4 changes: 2 additions & 2 deletions docs/examples/kubo-as-a-library/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,8 +298,8 @@ github.com/ipfs-shipyard/nopfs/ipfs v0.25.0 h1:OqNqsGZPX8zh3eFMO8Lf8EHRRnSGBMqcd
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0/go.mod h1:BxhUdtBgOXg1B+gAPEplkg/GpyTZY+kCMSfsJvvydqU=
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37 h1:q3a+2FIbWzZbx/yUqpuG4jLVSa6GvxtRfx9TU5GLiN0=
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37/go.mod h1:omQZmLS7LegSpBy3m4CrAB9/SO7Fq3pfv+5y1FOd+gI=
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161 h1:SfqEWWvIMYj+cbmtAFW7/oSrI0FGfs69jro5/w0sLig=
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161/go.mod h1:omQZmLS7LegSpBy3m4CrAB9/SO7Fq3pfv+5y1FOd+gI=
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
github.com/ipfs/go-bitswap v0.11.0 h1:j1WVvhDX1yhG32NTC9xfxnqycqYIlhzEzLXG/cU1HyQ=
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ require (
github.com/hashicorp/go-version v1.7.0
github.com/ipfs-shipyard/nopfs v0.0.14
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161
github.com/ipfs/go-block-format v0.2.0
github.com/ipfs/go-cid v0.5.0
github.com/ipfs/go-cidutil v0.1.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,8 +362,8 @@ github.com/ipfs-shipyard/nopfs/ipfs v0.25.0 h1:OqNqsGZPX8zh3eFMO8Lf8EHRRnSGBMqcd
github.com/ipfs-shipyard/nopfs/ipfs v0.25.0/go.mod h1:BxhUdtBgOXg1B+gAPEplkg/GpyTZY+kCMSfsJvvydqU=
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37 h1:q3a+2FIbWzZbx/yUqpuG4jLVSa6GvxtRfx9TU5GLiN0=
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37/go.mod h1:omQZmLS7LegSpBy3m4CrAB9/SO7Fq3pfv+5y1FOd+gI=
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161 h1:SfqEWWvIMYj+cbmtAFW7/oSrI0FGfs69jro5/w0sLig=
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161/go.mod h1:omQZmLS7LegSpBy3m4CrAB9/SO7Fq3pfv+5y1FOd+gI=
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
github.com/ipfs/go-bitswap v0.11.0 h1:j1WVvhDX1yhG32NTC9xfxnqycqYIlhzEzLXG/cU1HyQ=
Expand Down
2 changes: 1 addition & 1 deletion test/dependencies/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ require (
github.com/huin/goupnp v1.3.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37 // indirect
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161 // indirect
github.com/ipfs/go-bitfield v1.1.0 // indirect
github.com/ipfs/go-block-format v0.2.0 // indirect
github.com/ipfs/go-cid v0.5.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions test/dependencies/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,8 +298,8 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37 h1:q3a+2FIbWzZbx/yUqpuG4jLVSa6GvxtRfx9TU5GLiN0=
github.com/ipfs/boxo v0.29.2-0.20250415191135-dc60fe747c37/go.mod h1:omQZmLS7LegSpBy3m4CrAB9/SO7Fq3pfv+5y1FOd+gI=
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161 h1:SfqEWWvIMYj+cbmtAFW7/oSrI0FGfs69jro5/w0sLig=
github.com/ipfs/boxo v0.29.2-0.20250424075053-95aa3f00b161/go.mod h1:omQZmLS7LegSpBy3m4CrAB9/SO7Fq3pfv+5y1FOd+gI=
github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA=
github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU=
github.com/ipfs/go-block-format v0.2.0 h1:ZqrkxBA2ICbDRbK8KJs/u0O3dlp6gmAuuXUJNiW1Ycs=
Expand Down
Loading