v2.0.0

Overview

We're happy to announce the Boost v2.0.0 release. This release revamps the way Boost handles indices for user data, introducing the Local Index Directory (LID) and removing the DAG store.

The Local Index Directory requires a database for persistence. After reviewing various options and consulting the community, we are providing interfaces to two databases - LevelDB (for SPs holding less than 1 PiB of user data) and YugabyteDB (for SPs holding more than 1 PiB of user data). ❗ Please take your future growth into account before choosing a database ❗

👉 ⚠️ ❗ ❗ Before updating, please read the release notes and documentation carefully, as storage providers will need to setup a database service and go through a database migration process. Incorrect process can result in permanent data loss ❗ ❗ ⚠️

Highlights

Local Index Directory (LID)

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (CID).

Currently this task is performed by the DAG store component. The DAG store keeps its indices on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools. It makes it easy to understand what data is index and retrievable, and what data is archived and requires unsealing to make it retrievable.

Legacy storage deals are deprecated

Boost will no longer support legacy deals by default. Users will get an error message stating the same when they try to make deals using deal protocols older than mk2.0
To avoid breaking the deal flow, we have a new configuration option to enable them back. We highly recommend switching to new deals as this option will be removed in future releases.

[Dealmaking]
  # Whether to enable legacy deals on the Boost node or not. We recommend keeping
  # them disabled. These will be completely deprecated soon.
  #
  # type: bool
  #EnableLegacyStorageDeals = false

⚠️ Boost v2 documentation will now be default. Users can select v1 from the dropdown if they wish to refer to Boost v1 documentation.

⚠️ Boost will no longer be backporting any fixes to v1 releases

Upgrading to Boost v2

You can find detailed instructions on how to upgrade to Boost v2 via the Boost Documentation Tutorials - How to Migrate to Boost v2.

Full Changelog: v1.7.4...v2.0.0

v2.0.0-rc1

Overview

We're happy to announce the Boost v2.0.0-rc1 release. This release revamps the way Boost handles indices for user data, introducing the Local Index Directory (LID) and removing the DAG store.

The Local Index Directory requires a database for persistence. After reviewing various options and consulting the community, we are providing interfaces to two databases - LevelDB (for SPs holding less than 1 PiB of user data) and YugabyteDB (for SPs holding more than 1 PiB of user data). ❗ Please take your future growth into account before choosing a DB type ❗

👉 ⚠️ ❗ ❗ Before updating, please read the release notes and documentation carefully, as storage providers will need to setup a database service and go through a database migration process. Incorrect process can result in permanent data loss ❗ ❗ ⚠️

Highlights

Local Index Directory (LID)

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (CID).

Currently this task is performed by the DAG store component. The DAG store keeps its indices on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools. It makes it easy to understand what data is index and retrievable, and what data is archived and requires unsealing to make it retrievable.

Upgrading to Boost v2

You can find detailed instructions on how to upgrade to Boost v2 via the Boost Documentation Tutorials - How to Migrate to Boost v2.

Full Changelog: v1.7.3...v2.0.0-rc1

v1.7.4

Overview

We're happy to announce the Boost v1.7.4 release.

This release is compatible with the Lotus v1.23.0 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

Improvements

• Fixes the UI bug alerting about all stuck messages instead of local stuck messages

What's Changed

• feat: mpool monitor by @LexLuthr in #1512
• Add monitoring alert UI element by @dirkmc in #1517
• Add flag to allow restarting devnet by @lalexgap in #1521
• Update lotus and boxo versions by @dirkmc in #1466
• backport #1523 to main by @LexLuthr in #1556

New Contributors

• @lalexgap made their first contribution in #1521

Full Changelog: v1.7.3...v1.7.4

v1.7.3

Overview

We're happy to announce the Boost v1.7.3 release.
This release is compatible with the Lotus v1.23.0 and v.23.1-rcX release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

🌟 Highlights

Option to disable tagging of wallet funds

• By default, Boost tags funds for each deal with the value defined by MaxPublishDealsFee, and untags the funds when the deal is published. This is to prevent SPs from accepting too many deals and then not being able to publish them due to a lack of funds.
• Some SPs would prefer to manage their wallet funds themselves.
• Tagging can now be disabled by setting the following variable in Boost config

   	// Whether to enable tagging of funds. If enabled, each time a deal is
	// accepted boost will tag funds for that deal so that they cannot be used
	// for any other deal.
	FundsTaggingEnabled bool

HTTP announcements to the IPNI

• Boost can now announce advertisements over HTTP to the IPNI
• Once enabled, it should reduce the request load over Graphsync and improve performance of data retrievals
• New configuration to enable HTTP announcements

  PurgeCacheOnStart = true
  
    [IndexProvider.Announce]
      # Make a direct announcement to a list of indexing nodes over http.
      # Note that announcements are already made over pubsub regardless
      # of this setting.
      #
      # type: bool
      # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_ANNOUNCEOVERHTTP
      AnnounceOverHttp = true
  
      # The list of URLs of indexing nodes to announce to.
      #
      # type: []string
      # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_DIRECTANNOUNCEURLS
      DirectAnnounceURLs = ["https://cid.contact/ingest/announce", "http://localhost:3001"]
  
    [IndexProvider.HttpPublisher]
      # If not enabled, requests are served over graphsync instead.
      #
      # type: bool
      # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_ENABLED
      #Enabled = true
  
      # Set the public hostname / IP for the index provider listener.
      # eg "82.129.73.111"
      # This is usually the same as the for the boost node.
      #
      # type: string
      # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PUBLICHOSTNAME
      #PublicHostname = "xx.xx.xx.xx"
  
      # Set the port on which to listen for index provider requests over HTTP.
      # Note that this port must be open on the firewall.
      #
      # type: int
      # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PORT
     # Port = 3104
  

Fix for stalled Graphsync retrievals

• When a deal is announced, the network indexer queries boost to get the cids of each block in the deal data
• There was an issue with the code that could cause retrievals to stall.
• Requests made via the network indexers, like cid.contact, are now routed through the simplified graphsync retrieval pathways as opposed to the legacy retrievals which is prone to stalling, and is less performant.
• Stalled Graphsync retrieval cancellation code has been refactored for faster execution and efficiency

🏗️ Improvements

Managing Mpool

• Deal page will display the number of epochs elapsed since PSD message was sent. This should help SPs to quickly find PSD messages stuck in the local mpool by looking at the number of epochs elapsed vs message confidence (10 epochs)
• Mpool section in WebUI displays the method parameters for each message

Improved configuration and defaults for internet interactions

• Users can specify a listen address for booster-http instead of listening on all available addresses
• WebUI now listens on localhost by default to avoid accidentally exposing it to the internet

Deal making

• Better deal-acceptance timeout implementation to avoid timeout due to the execution time of the external deals filters
• A listen address can be provided for GraphQL service. This should mitigate security risks for publicly exposed Boost nodes.
• Offline deals waiting for data can be cancelled from the UI.
• Boost will now fail the deal if a snap-deal sector reverts to Proving at any point of the sealing process. Earlier, Boost would incorrectly mark such deals as successfully sealed.
• Boost now supports local commp calculation for 64 GiB sectors.

Others

• Indexer should be able to skip bad advertisements when processing the Ad chain from a Boost node
• All the available Boost APIs can now be listed using rpc.discover or Filecoin.Discover method
• Boost now has a config command similar to lotus config command with additional flag --diff to generate a diff between current and default config

📣 Deprecation Notice: Disabling payments via built in payment channels actor in Boost 📣

The Boost team will be disabling payments via built in payment channels actor by default. There has been limited activity on these payment channels on the entire network. We are planning on disabling this in an effort to simplify the code base for existing retrieval use cases and improve retrieval reliability.

A few items to note:

• The main use case for payment channels are for micropayments where small payments are made as data is sent back. With payment channels you do not need to keep sending messages to mainnet.
• There is a PLN team actively working on payment channels, and is starting to look into integration with the Filecoin network.
• You can still make payments for retrieval deals on mainnet, but you will need to negotiate payment terms for a retrieval deal with a SP directly.

What to expect:

1. Boost maintainers will update default behavior to use new retrieval code (optimized for retrievals without payment channels).
2. Please report any issues you have in #boost-help, so we can track and monitor SP impact.
3. If you have any questions or concerns, please raise them in Boost discussion, or reach out in #boost-help

What's Changed

• initial yugabyte dockerfile by @nonsense in #1393
• Add a config flag to disable funds tagging by @dirkmc in #1367
• chore:add graphql server listen address config by @rickiey in #1327
• chore: update lotus and ffi versions (#1400) by @jacobheun in #1401
• chore: fix docsgen on main by @jacobheun in #1402
• put g.activeRetrievals behind a lock everywhere by @nonsense in #1404
• chore: update lotus and ffi versions (#1400) by @jacobheun in #1409
• fix: mpool all messages panic by @LexLuthr in #1405
• feat: add cancel button for offline deals waiting for data by @LexLuthr in #1413
• chore: enable files gateway in devnet by default by @jacobheun in #1420
• car size required only for online deals by @anjor in #1421
• Dont throw error if deal already announced by @dirkmc in #1426
• fix: use new retrieval code path for indexers by @jacobheun in #1425
• fix usage comments on boostd data-transfers commands by @dirkmc in #1437
• fix download block link spacing by @dirkmc in #1438
• fix: fail deal if snap-deal sectors fails sealing by @LexLuthr in #1419
• fix: 64 GiB local commp by @LexLuthr in #1439
• chore: release v1.7.3-rc1 by @LexLuthr in #1441
• Update README.md by @LaurenSpiegel in #1443
• feat: enable listing API methods for Boost by @LexLuthr in #1442
• fix: statx output string by @LexLuthr in #1451
• fix flaky TestMultipleDealsConcurrent by @dirkmc in #1458
• Add option to serve index provider ads over http by @dirkmc in #1452
• Signal to index provider to skip announcements by @dirkmc in #1457
• chore: release v1.7.3-rc2 by @LexLuthr in #1460
• fix: improve stalled retrieval cancellation by @LexLuthr in #1449
• feat: enable listen address for booster-http by @LexLuthr in #1461
• fix: nil pointer error for all messages in UI by @LexLuthr in #1470
• fix: incorrect check when import offline deal data using proposal CID by @LexLuthr in #1473
• Properly cancel graphsync requests in unpaid retrieval by @hannahhoward in #1475
• fix: set UI default listen address to localhost by @LexLuthr in h...

v1.7.3-rc3

Overview

We're happy to announce the Boost v1.7.3-rc3 release.

📣 Deprecation Notice: Disabling payments via built in payment channels actor in Boost 📣

The Boost team will be disabling payments via built in payment channels actor by default. There has been limited activity on these payment channels on the entire network. We are planning on disabling this in an effort to simplify the code base for existing retrieval use cases and improve retrieval reliability.

A few items to note:

• The main use case for payment channels are for micropayments where small payments are made as data is sent back. With payment channels you do not need to keep sending messages to mainnet.
• There is a PLN team actively working on payment channels, and is starting to look into integration with the Filecoin network.
• You can still make payments for retrieval deals on mainnet, but you will need to negotiate payment terms for a retrieval deal with a SP directly.

What to expect:

1. Boost maintainers will update default behavior to use new retrieval code (optimized for retrievals without payment channels).
2. Please report any issues you have in #boost-help, so we can track and monitor SP impact.
3. If you have any questions or concerns, please raise them in Boost discussion, or reach out in #boost-help

This release is compatible with the Lotus v1.23.0 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

Improvements

Managing Mpool

• Deal page will display the number of epochs elapsed since PSD message was sent. This should help SPs to quickly find PSD messages stuck in the local mpool by looking at the number of epochs elapsed vs message confidence (10 epochs)
• Mpool section in WebUI displays the method parameters for each message

Improved configuration and defaults for internet interactions

• Users can specify a listen address for booster-http instead of listening on all available addresses
• WebUI now listens on localhost by default to avoid accidentally exposing it to the internet

Deal making

• Better deal-acceptance timeout implementation to avoid timeout due to the execution time of the external deals filters

What's Changed

• feat: enable listen address for booster-http by @LexLuthr in #1461
• fix: nil pointer error for all messages in UI by @LexLuthr in #1470
• fix: incorrect check when import offline deal data using proposal CID by @LexLuthr in #1473
• Properly cancel graphsync requests in unpaid retrieval by @hannahhoward in #1475
• fix: set UI default listen address to localhost by @LexLuthr in #1476
• feat: display msg params in the mpool UI by @LexLuthr in #1471
• Reset read deadline after reading deal proposal message by @dirkmc in #1479
• feat: Show elapsed epoch and PSD wait epochs in UI by @LexLuthr in #1480
• chore: release v1.7.3-rc3 by @LexLuthr in #1481

Full Changelog: v1.7.3-rc2...v1.7.3-rc3

v1.7.3-rc2

Overview

We're happy to announce the Boost v1.7.3-rc2 release.

📣 Deprecation Notice: Disabling payments via built in payment channels actor in Boost 📣

The Boost team will be disabling payments via built in payment channels actor by default. There has been limited activity on these payment channels on the entire network. We are planning on disabling this in an effort to simplify the code base for existing retrieval use cases and improve retrieval reliability.

A few items to note:

• The main use case for payment channels are for micropayments where small payments are made as data is sent back. With payment channels you do not need to keep sending messages to mainnet.
• There is a PLN team actively working on payment channels, and is starting to look into integration with the Filecoin network.
• You can still make payments for retrieval deals on mainnet, but you will need to negotiate payment terms for a retrieval deal with a SP directly.

What to expect:

1. Boost maintainers will update default behavior to use new retrieval code (optimized for retrievals without payment channels).
2. Please report any issues you have in #boost-help, so we can track and monitor SP impact.
3. If you have any questions or concerns, please raise them in Boost discussion, or reach out in #boost-help

This release is compatible with the Lotus v1.23.0 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

🌟 Highlights

HTTP announcements to the IPNI

• Boost can now announce advertisements over HTTP to the IPNI
• Once enabled, it should reduce the request load over Graphsync and improve performance of data retrievals
• New configuration to enable HTTP announcements

  PurgeCacheOnStart = true
  
    [IndexProvider.Announce]
      # Make a direct announcement to a list of indexing nodes over http.
      # Note that announcements are already made over pubsub regardless
      # of this setting.
      #
      # type: bool
      # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_ANNOUNCEOVERHTTP
      AnnounceOverHttp = true
  
      # The list of URLs of indexing nodes to announce to.
      #
      # type: []string
      # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_DIRECTANNOUNCEURLS
      DirectAnnounceURLs = ["https://cid.contact/ingest/announce", "http://localhost:3001"]
  
    [IndexProvider.HttpPublisher]
      # If not enabled, requests are served over graphsync instead.
      #
      # type: bool
      # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_ENABLED
      #Enabled = true
  
      # Set the public hostname / IP for the index provider listener.
      # eg "82.129.73.111"
      # This is usually the same as the for the boost node.
      #
      # type: string
      # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PUBLICHOSTNAME
      #PublicHostname = "xx.xx.xx.xx"
  
      # Set the port on which to listen for index provider requests over HTTP.
      # Note that this port must be open on the firewall.
      #
      # type: int
      # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PORT
     # Port = 3104
  

Improvements

• Stalled Graphsync retrieval cancellation code has been refactored for faster execution and efficiency
• Indexer should be able to skip bad advertisements when processing the Ad chain from a Boost node
• All the available Boost APIs can now be listed using rpc.discover or Filecoin.Discover method

What's Changed

• feat: enable listing API methods for Boost by @LexLuthr in #1442
• Add option to serve index provider ads over http by @dirkmc in #1452
• Signal to index provider to skip announcements by @dirkmc in #1457
• fix: improve stalled retrieval cancellation by @LexLuthr in #1449

Full Changelog: v1.7.3-rc1...v1.7.3-rc2

v1.7.3-rc1

Overview

We're happy to announce the Boost v1.7.3-rc1 release.

⚠️ The minimum required Golang version for Boost is now v1.19.7, and Golang 1.20.x is Highly Recommended due to known potential panics ⚠️

Golang versions are now included in the compatibility matrix.

This release is compatible with the Lotus v1.23.0 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

🌟 Highlights

Option to disable tagging of wallet funds

• By default, Boost tags funds for each deal with the value defined by MaxPublishDealsFee, and untags the funds when the deal is published. This is to prevent SPs from accepting too many deals and then not being able to publish them due to a lack of funds.
• Some SPs would prefer to manage their wallet funds themselves.
• Tagging can now be disabled by setting the following variable in Boost config

   	// Whether to enable tagging of funds. If enabled, each time a deal is
	// accepted boost will tag funds for that deal so that they cannot be used
	// for any other deal.
	FundsTaggingEnabled bool

Fix for stalled Graphsync retrievals

• When a deal is announced, the network indexer queries boost to get the cids of each block in the deal data
• There was an issue with the code that could cause retrievals to stall.
• Requests made via the network indexers, like cid.contact, are now routed through the simplified graphsync retrieval pathways as opposed to the legacy retrievals which is prone to stalling, and is less performant.

Improvements

• A listen address can be provided for GraphQL service. This should mitigate security risks for publicly exposed Boost nodes.
• Offline deals waiting for data can be cancelled from the UI.
• Boost will now fail the deal if a snap-deal sector reverts to Proving at any point of the sealing process. Earlier, Boost would incorrectly mark such deals as successfully sealed.
• Boost now supports local commp calculation for 64 GiB sectors.

What's Changed

• Add a config flag to disable funds tagging by @dirkmc in #1367
• chore:add graphql server listen address config by @rickiey in #1327
• put g.activeRetrievals behind a lock everywhere by @nonsense in #1404
• fix: mpool all messages panic by @LexLuthr in #1405
• feat: add cancel button for offline deals waiting for data by @LexLuthr in #1413
• chore: enable files gateway in devnet by default by @jacobheun in #1420
• car size required only for online deals by @anjor in #1421
• Don't throw error if deal already announced by @dirkmc in #1426
• fix: use new retrieval code path for indexers by @jacobheun in #1425
• fix usage comments on boostd data-transfers commands by @dirkmc in #1437
• fix download block link spacing by @dirkmc in #1438
• fix: fail deal if snap-deal sectors fails sealing by @LexLuthr in #1419
• fix: 64 GiB local commp by @LexLuthr in #1439

New Contributors

• @rickiey made their first contribution in #1327

Full Changelog: v1.7.2...v1.7.3-rc1

v1.7.2

Overview

We're happy to announce the Boost v1.7.2 release. This is a direct upgrade from 1.7.1 and 1.7.0, and fixes non critical but noisy 'unknown actor code' logs as a result of the mainnet upgrade.

This is a compatibility release is for the Lotus v1.23.0 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

What's Changed

• chore: update lotus and ffi versions by @jacobheun in #1400

Full Changelog: v1.7.1...v1.7.2

v1.6.4

Overview

We're happy to announce the Boost v1.6.4 release. This is a direct upgrade from 1.6.3, and fixes non critical but noise 'unknown actor code' logs as a result of the mainnet upgrade.

This is a compatibility release is for the Lotus v1.22.1 release. See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

What's Changed

• chore: bump lotus and ffi versions by @jacobheun in #1399

Full Changelog: v1.6.3...v1.6.4

v1.7.1

Overview

We're happy to announce the Boost v1.7.1 patch release.

⚠️ The minimum required Golang version for Boost is now v1.19.7, and Golang 1.20.x is Highly Recommended due to known potential panics ⚠️

Golang versions are now included in the compatibility matrix.

Lotus v1.23.0 Compatibility

This release is NOT required for the mainnet upgrade, but if you are running Boost 1.7.0 this will fix an issue with index announcements that can prevent retrieval from working properly.
See the Boost and Lotus compatibility matrix for up to date information, and to plan your upgrade.

Please make sure to upgrade to test the new features shipped in this release. See our upgrade guide for details.

Legacy deals support in Boost - EOL notice

The Boost team will be suspending support for legacy deals, as legacy markets capabilities now fully deprecated (lotus-markets EOL as of Jan 31, 2023). Furthermore, most aggregators in the Filecoin network such as Estuary, dotStorage, etc. have moved to using Boost for storage dealmaking. Other large clients are also working with SPs using Boost.

We recommend SPs and their clients, as well as other users, to move to using Boost for storage dealmaking as soon as possible. Here is a tentative timeline for what to expect:

• Boost maintainers will be fully suspending support for legacy deals in Boost starting May 15, 2023.
• If you have any questions or concerns, please raise them in Boost discussion, or reach out in #boost-help on Filecoin Slack

🌟 Highlights

🐞 Bug Fix: Prevent accidental removal of valid sector index announcements

• We discovered a 🐞 bug 🐛 where when calling the Lotus Storage API we were not properly handling the multiple states that can be returned for any given sector (Cache, Unsealed, Sealed, etc). This could result in a sector having the wrong state recorded, including treating it as removed for indexing purposes.
• This fix includes a database migration to fix an index issue in the SectorStatus database of boost. This migration is quick, but will result in a full ingest of index announcements for users coming from 1.7.0.

What's Changed

• fix: prevent accidental removal of valid sector index announcements by @jacobheun in #1396

Full Changelog: v1.7.0...v1.7.1