v0.20.3-mocha

This patch release fixes a JSON serialisation error with libshare.Blob via #3958.

What's Changed

• refactor: remove blob size check by @rootulp in #3934
• fix(core): change fill block mode by @cristaloleg in #3936
• hotfix(core): don't hold on Data by @Wondertan in #3926
• fix(p2p): disable quic by @walldiss in #3937
• chore(deps): temporary fork boxo by @Wondertan in #3939
• refactor(share/availability): Move window and constants to share/availability pkg by @renaynay in #3906
• fix(shwap/bitswap): tune bitswap params by @Wondertan in #3940
• fix(shwap/bitswap): Blockstore.GetSize: getting size with no compute by @Wondertan in #3894
• chore(logs): cleanup logging by @Wondertan in #3945
• fix(shwap/bitswap): update boxo and config by @Wondertan in #3946
• chore(go.mod): bump boxo fork by @Wondertan in #3948
• fix(share/availability/light): Allow retries on ODS size 1 by @renaynay in #3944
• perf(shwap/bitswap): add session pools inside Getter by @Wondertan in #3947
• chore(deps): Bump cosmossdk.io/math from 1.3.0 to 1.4.0 by @dependabot in #3955
• chore(go.mod) bump go-square by @vgonkivs in #3958

Full Changelog: v0.20.2-mocha...v0.20.3-mocha

v0.20.3-arabica

This patch release fixes a JSON serialisation error with libshare.Blob via #3958.

What's Changed

• refactor: remove blob size check by @rootulp in #3934
• fix(core): change fill block mode by @cristaloleg in #3936
• hotfix(core): don't hold on Data by @Wondertan in #3926
• fix(p2p): disable quic by @walldiss in #3937
• chore(deps): temporary fork boxo by @Wondertan in #3939
• refactor(share/availability): Move window and constants to share/availability pkg by @renaynay in #3906
• fix(shwap/bitswap): tune bitswap params by @Wondertan in #3940
• fix(shwap/bitswap): Blockstore.GetSize: getting size with no compute by @Wondertan in #3894
• chore(logs): cleanup logging by @Wondertan in #3945
• fix(shwap/bitswap): update boxo and config by @Wondertan in #3946
• chore(go.mod): bump boxo fork by @Wondertan in #3948
• fix(share/availability/light): Allow retries on ODS size 1 by @renaynay in #3944
• perf(shwap/bitswap): add session pools inside Getter by @Wondertan in #3947
• chore(deps): Bump cosmossdk.io/math from 1.3.0 to 1.4.0 by @dependabot in #3955
• chore(go.mod) bump go-square by @vgonkivs in #3958

Full Changelog: v0.20.2-arabica...v0.20.3-arabica

v0.20.2

NOTE: v0.18 and v0.19 are skipped for mainnet

Hey everyone, this is quite a mammoth of a release, so please hang tight. We have two major changes packaged in this release:

• Shwap (protocol-breaking)
• Celestia v3 compatibility (API-breaking)

The migration period for this release is quite tight (a bit longer than 2 weeks), so we urge all node operators to upgrade to this version and re-sync against the network by December 2nd.

As this release is API-breaking, we also urge all integrations to migrate to this new version by December 2nd. (The list of API breaks can be found below).

Shwap

This release features Shwap(CIP-19), a protocol-breaking feature that brings massive optimizations to disk usage(16x) and data access latencies. The new protocol is not backwards compatible with the currently deployed protocol and software.

Both protocols will run side-by-side for about a two-week transition period, from November 20 to December 2. During this period, node operators should migrate to the new version. At the end of the period, posted historical and recent data will no longer be available on the old protocol.

Migration Notes

Migration requires resyncing the node from scratch, as it now contains a new storage format.

Migration Steps over the old node:

• Remove old data:
  • In home directory(default .celestia-<type>) => rm -rf inverted_index index data transients blocks
  • Keep keys directory to preserve peer and chain identity keys in place
• Config updates
  • If defaults were in use, remove the old config and re-init in the next step
  • If you want to preserve the old config - move BlockstoreCacheSize entry from [Share.EDSStoreParams] section to just [Share]and remove GCInterval, RoutingTableRefreshPeriod
• Re-init node for the Mocha p2p network
  • celestia init
• Start the node and monitor syncing
  • celestia start

If the old node has to be kept running - initiate and start the new version in a different directory:

• celestia init --node.store <new_path>
• celestia start --node.store <new_path>

Celestia V3 compatibility

This release contains an upgrade to the celestia-app version (to v3). v3 will only be live on the mainnet in mid-December, but this release contains the requirements for compatibility with the upgrade.

Notable API breaks include:

• simplification of the blob share and state modules via the deduplication of types (via [#3782](#3782))
• share module now takes height instead of the full ExtendedHeader (via [#3870](#3870))

Comprehensive list of API breaks:

• GetSharesByNamespace renamed to GetNamespaceData
• the share module returns shwap.NamespaceData for the GetNamespaceData method
• the share module now takes height (uint64) instead of the full ExtendedHeader for all methods
• both the share and blob modules now use the go-square v2 libshare.Namespace (from https://github.com/celestiaorg/go-square/tree/v2.0.0) instead of the old share.Namespace for all methods
• the state module now uses the go-square v2 libshare.Blob type (from https://github.com/celestiaorg/go-square/tree/v2.0.0) instead of the old one from the state module
• the blob package now embeds the new go-square v2 libshare.Blob type mentioned above in the blob.Blob struct instead of the old v1 type.

Miscellaneous

What to look out for in the following releases:

• the -experimental-pruning flag will soon be deprecated in favour of pruning becoming the default mode for all node types, requiring users to pass -archival to retain block history beyond the sampling window
• the API gateway will be deprecated

What's Changed

Features

• feat(shwap): Add eds store by @walldiss in #3545
• feat(shwap): Add namespace data shwap id by @walldiss in #3587
• feat(shwap): Add axis roots to Accessor by @walldiss in #3586
• feat(shwap): Add bufferisation for writing ods/Q1Q4 files by @walldiss in #3592
• feat(shwap/p2p): Bitswap Getter by @Wondertan in #3584
• feat(shwap): Shwap add readFrom Writeto methods to shwap ids by @walldiss in #3605
• feat(shwap): Integrate shrex into shwap by @walldiss in #3554
• feat(shwap): Shwap core integration by @walldiss in #3598
• feat(shwap): Shwap cascade and mock getters by @walldiss in #3599
• feat(shwap): Update availability to work with shwap by @walldiss in #3600
• feat(shwap): add shwap to blob module by @walldiss in #3613
• feat(shwap): update daser for shwap by @walldiss in #3614
• feat(shwap): add shwap store getter by @walldiss in #3617
• feat(shwap): Add caching to blockstore by @walldiss in #3615
• feat(shwap/bitswap): Shwap optimized Bitswap constructors by @Wondertan in #3536
• feat(shwap): Improve eds store cache refcounter forgot Close() protection by @walldiss in #3638
• feat(shwap): shwap nodebuilder and cleanup by @walldiss in #3637
• feat(shwap/ods_file): tail padding trim by @Wondertan in #3620
• feat(shwap): Add discovery version by @walldiss in #3616
• feat(cmd/cel-shed): shwap by @Wondertan in #3720
• feat(shwap/store): Add q4 trimming support in store by @walldiss in #3680
• feat(nodebuilder): Blockstore cache params by @walldiss in #3739
• feat(share/availability): Dont store q4 on archival by @walldiss in #3751
• feat(store/cache): add Has methods by @Wondertan in #3757
• feat(cel-shed/p2p): bootstrapper probe tool by @smuu in #3780
• feat!: Live, laugh, Shwap by @Wondertan in #3675
• feat(cmd): adding timeout flag by @sebasti810 in #3580
• feat(nodebuilder/da): implement go-da v0.6.1 changes by @tzdybal in #3750
• feat(modp2p): close peer connections after successful blocking by @notlelouch in #3838
• feat(shwap/bitswap): use new option for optimized Has check by @Wondertan in #3813
• feat(store): Test all put funcs by @walldiss in #3847
• feat(store): use use relative path for empty blocks symlinks by @walldiss in #3861
• feat(store): correct corrupted files on write by @walldiss in #3859
• feat(share/availability)!: Store success results for light availability by @walldiss in #3887
• feat(pruner/light): add light pruning for shwap by @walldiss in #3896

Fixes

• fix(shwap/bitswap): close acccessor in Blockstore by @Wondertan in #3535
• fix(shwap/store): ensure EDS is accessible during writes by @Wondertan in #3596
• fix(shwap): don't cache empty eds and close recent accessor by @walldiss in #3608
• fix(store/files): ODSQ4 Closing audit by @Wondertan in #3691
• fix(shwap/cahce): use local copy of axis in proofs cache by @walldiss in #3657
• fix(shwap): io.ReadFull for read helpers by @Wondertan in #3696
• fix(shwap): io.ReadFull for NamespaceDataID by @walldiss in #3697
• fix(shwap/bitswap): correct protocol id by @Wondertan in #3722
• fix(store/odsq4): fix order of opened flag stored by @walldiss in #3735
• fix(store/ods): fix full ods caching by @walldiss in #3736
• fix(core): fix flaky generateNonEmptyBlocks by @walldiss in #3740
• fix(blob): fix blobsub subscription cancellation test by @walldiss in #3746
• fix(store): use symlinks for empty blocks heighs by @walldiss in #3747
• fix(header): soft failure is only for not enough voting power by @cristaloleg in #3769...

v0.20.2-mocha

This patch brings numerous fixes to the sampling protocol, particularly Bitswap sub-protocol dependency, by fixing 7(!) client-side memory leaks. It also improves resource utilization and load balancing on the server side. All the changes are currently applied in our fork(https://github.com/celestiaorg/boxo/tree/fixes) and will be upstreamed with unforking in an upcoming release.

Additionally, the patch applies a safety measure by temporarily disabling QUIC due to the unknown origin of the following bug: quic-go/quic-go#4712.

As a cherry on top, this patch hotfixes long-standing memory issues in BNs: #3926. Thanks to @rach-id for helping us with the debugging.

Full Changelog: v0.20.1-mocha...v0.20.2-mocha

v0.20.2-arabica

This patch brings numerous fixes to the sampling protocol, particularly Bitswap sub-protocol dependency, by fixing 7(!) client-side memory leaks. It also improves resource utilization and load balancing on the server side. All the changes are currently applied in our fork(https://github.com/celestiaorg/boxo/tree/fixes) and will be upstreamed with unforking in an upcoming release.

Additionally, the patch applies a safety measure by temporarily disabling QUIC due to the unknown origin of the following bug: quic-go/quic-go#4712.

As a cherry on top, this patch hotfixes long-standing memory issues in BNs: #3926. Thanks to @rach-id for helping us with the debugging.

Full Changelog: v0.20.1-arabica...v0.20.2-arabica

v0.20.1-mocha

What's Changed

• deps(go.mod): bump app to arabica tag by @renaynay in #3910
• chore: bump to celestia-app v3.0.0-mocha by @evan-forbes in #3920

Full Changelog: v0.20.0-mocha...v0.20.1-mocha

v0.20.1-arabica

What's Changed

• ci(docker): bump version by @tty47 in #3914
• deps(go.mod): bump app to arabica tag by @renaynay in #3910

Full Changelog: v0.20.0-arabica...v0.20.1-arabica

v0.20.0-mocha

It's the equivalent of https://github.com/celestiaorg/celestia-node/releases/tag/v0.20.0-arabica but for Mocha and with a docker fix.

What's Changed

• ci(docker): bump version by @tty47 in #3914

Full Changelog: v0.20.0-arabica...v0.20.0-mocha

v0.20.0-arabica

This release breaks APIs and storage. The API break renames a method(#3902), grouping it closer to breaks in v0.19.0.

The storage break affects only LNs (#3887) and is necessary to support LN pruning post-Shwap. LN runners must resync their nodes. See the docs for further instruction.

Besides, this release brings multiple preventive improvements to sampling logic.

What's Changed(since v0.19.0-arabica)

Breaking

• !refactor: rename GetSharesByNamespace to GetNamespaceData by @cristaloleg in #3902
• feat(share/availability)!: Store success results for light availability by @walldiss in #3887

Feat

• feat(pruner/light): add light pruning for shwap by @walldiss in #3896

Fixes

• fix(pruner): fix find algo so that it does not pass a malformed range into GetRangeByHeight call by @renaynay in #3828
• fix: wrong network error by @jcstein in #3885
• fix(share/availability): propagate errors from light availability properly by @walldiss in #3888
• fix(shwap/shrex): enable recovery middleware in shrex server by @walldiss in #3897
• fix(DASer): Log sampling errors in DASer by @walldiss in #3889
• fix(availability): prevent parallel availability calls by @walldiss in #3883
• fix(ci): GH_TOKEN for upload-docs by @cristaloleg in #3877
• fix(ci): do not version_bump by @cristaloleg in #3882

Chores

• chore(share): remove unused ShareWithProof by @cristaloleg in #3893
• chore: remove unused code by @walldiss in #3898
• test(swamp): cover v1 blobs in swamp tests by @vgonkivs in #3908
• refactor(share/availability): simplify light availability by @walldiss in #3895

Full Changelog: v0.19.0-arabica...v0.20.0-arabica

v0.19.0-arabica

This release is API-breaking and contains an upgrade to the celestia-app version (to v3).

Notable API breaks include:

• deduplication of types that resulted in several type changes in the blob, share and state modules (via #3782)
• share module now takes height instead of the full ExtendedHeader (via #3870)

What to look out for in following releases:

• the --experimental-pruning flag will soon be deprecated in favour of pruning becoming the default mode for all node types, requiring users to pass --archival in order to retain block history beyond the sampling window
• the API gateway will be deprecated

What's Changed

• fix(nodebuilder): flag parsing for --node.type needs lowercase string, list network aliases for --p2p.network by @renaynay in #3827
• fix(share/p2p/bitswap): lower bitswap log severity by @walldiss in #3831
• chore(go.mod): Bump celestia-app to v2.2.0 release in main by @renaynay in #3826
• fix(store): check if link exists on write by @walldiss in #3844
• chore: remove some TODOs by @cristaloleg in #3810
• chore: bump to v2.3.0 by @evan-forbes in #3854
• fix(eds/cache): Fix prealloc for proof nodes by @walldiss in #3860
• feat(modp2p): close peer connections after successful blocking by @notlelouch in #3838
• feat(shwap/bitswap): use new option for optimized Has check by @Wondertan in #3813
• feat(store): Test all put funcs by @walldiss in #3847
• feat(store): use use relative path for empty blocks symlinks by @walldiss in #3861
• fix(store/cache): unstick cache metrics names by @walldiss in #3857
• fix(docs) update txResponse in docs by @walldiss in #3848
• refactor(store): write actual share size to ods by @walldiss in #3863
• fix(store/file): wait for q4 to be written before returning by @walldiss in #3865
• feat(store): correct corrupted files on write by @walldiss in #3859
• fix(ci,release): run on any new tag or trigger manually by @cristaloleg in #3855
• chore(ci): remove release tracking epic ci by @MSevey in #3866
• fix(share/ipld): use bits.Len64 instead of math.Log2 in newProofCollector by @notlelouch in #3867
• refactoring!(share): migrating to go-square v2 types by @vgonkivs in #3782
• chore(go.mod): bump header by @renaynay in #3872
• fix(nodebuilder/p2p): lint by @cristaloleg in #3875
• refactor(nodebuilder): add archival flag by @walldiss in #3868
• refactor(api): add gateway deprecation notice by @walldiss in #3869
• !refactor(api/share): use height in share module by @walldiss in #3870

New Contributors

• @notlelouch made their first contribution in #3838

Full Changelog: v0.18.2-arabica...v0.19.0-arabica