Skip to content

WIP: NOT TO BE MERGED! #1870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 58 commits into
base: v0.38.x-celestia
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
aaac0e5
chore: add recovery message types (#1608)
evan-forbes Feb 17, 2025
d097543
chore: block propagation reactor initial boilerplate (#1616)
rach-id Feb 17, 2025
8bd5e30
fix: silly CI
evan-forbes Feb 19, 2025
d546bf5
chore: revert
evan-forbes Feb 19, 2025
294a467
feat: Add the ability to add parity data (#1618)
evan-forbes Feb 20, 2025
7c64aeb
feat: add proposal cache implementation (#1621)
rach-id Feb 20, 2025
2a46dcb
feat: add the peer state implementation (#1622)
rach-id Feb 20, 2025
461968e
feat: add the backbone of the propagation logic (#1623)
rach-id Feb 20, 2025
0496513
chore!: rewire where the compact block goes (#1625)
evan-forbes Feb 20, 2025
1a74806
chore!: rewire where the compact block goes (#1625) with some fixes
evan-forbes Feb 20, 2025
ea2ecca
feat: support have and want propagation (#1626)
rach-id Feb 20, 2025
b99653f
chore: add relevant TODOs
rach-id Feb 20, 2025
d49185f
feat: add support for recovery part (#1627)
rach-id Feb 21, 2025
a4dffc9
chore: add go routines management TODO
rach-id Feb 21, 2025
86c72f6
feat: define the propagator and use it in the state and consensus rea…
rach-id Feb 21, 2025
bdccdd3
feat: gossip the proposal (#1628)
evan-forbes Feb 21, 2025
e0b8401
chore: add TODO
rach-id Feb 21, 2025
f4a259b
chore: update the code owners in feature branch (#1634)
rach-id Feb 25, 2025
bc4499e
refactor!: clean up entire repo by putting the proposal in the compac…
evan-forbes Feb 25, 2025
88ac760
feat!: gossip the compact block (#1637)
evan-forbes Feb 28, 2025
742e57e
feat: broadcast haves when proposing a block
rach-id Mar 1, 2025
0fe2cc5
feat: support catchup
rach-id Mar 1, 2025
d90076c
chore: lint
rach-id Mar 1, 2025
acfc3c4
chore: gofumpt
rach-id Mar 1, 2025
3e341b2
feat: save the compact block when proposing and send it when connecti…
rach-id Mar 3, 2025
8dc3c78
feat!: reconstruct the block during recovery (#1652)
evan-forbes Mar 5, 2025
153fd24
feat: support recovering parts from the mempool (#1664)
rach-id Mar 12, 2025
548b30a
feat: verify the part hash during mempool recovery (#1670)
rach-id Mar 12, 2025
54257ea
feat!: use partshashes to create and verify proofs/roots for partset …
evan-forbes Mar 17, 2025
b5d501e
nits: cosmetics (#1690)
varun-doshi Mar 24, 2025
927e4dc
fix: enable new reactor in the e2e test (#1657)
evan-forbes Mar 24, 2025
0a944fb
chore: use cachedTx in mempool (#1680)
rach-id Mar 25, 2025
817640a
fix!: retrieve transactions from the mempool async, ignore irelevant …
evan-forbes Apr 2, 2025
c144771
feat: run the catchup periodically (#1707)
rach-id Apr 4, 2025
58d71de
feat: handle peer disconnections in the propagation reactor (#1685)
rach-id Apr 4, 2025
fa7f65d
feat: add support for compact block sign bytes (#1681)
rach-id Apr 4, 2025
c7e3508
fix: start block prop processing correctly + fix height relevance (#1…
rach-id Apr 10, 2025
a3837f2
feat: verify the proposal in the propagation reactor (#1689)
rach-id Apr 11, 2025
ba89c01
fix: panics in the propagation reactors (#1709)
evan-forbes Apr 20, 2025
2874493
fix: add defensive nil checks during msg type casting (#1772)
ninabarbakadze Apr 28, 2025
e6c0f9a
feat: verify haves parts hashes (#1779)
rach-id Apr 28, 2025
bb94711
chore: check if the tx metadata has overlapping ranges (#1783)
rach-id Apr 29, 2025
f148741
chore: return if compact block is nil in handle have (#1784)
rach-id Apr 29, 2025
2ad34b8
feat: limit per peer per block concurrent requests (#1773)
rach-id Apr 29, 2025
dc8c266
chore: implement RecoveryPart.ValidateBasics() (#1786)
rach-id Apr 29, 2025
97bbc54
chore: validate the recovery parts (#1788)
rach-id Apr 29, 2025
ba509ac
chore: trigger catchup when receiving a commitment (#1791)
rach-id Apr 30, 2025
171698c
chore: remove some dangling todos (#1790)
rach-id Apr 30, 2025
926b0a2
chore: correctly start the request from peer routine (#1789)
rach-id May 1, 2025
3bf9a57
chore: use the parts request limit in catchup (#1792)
rach-id May 1, 2025
52ad41f
fix: increment the current height when adding a commitment (#1793)
evan-forbes May 1, 2025
716d0c9
chore: check peer is still connected before writing to channel (#1810)
rach-id May 5, 2025
aec4908
feat: add the ability to sign over p2p messages and sign over the com…
evan-forbes May 9, 2025
bbce545
fix: adding the missing request in catchup correctly (#1818)
rach-id May 12, 2025
646728a
chore: verifying compact block height/round (#1859)
rach-id May 16, 2025
31d187e
chore: necessary cleanup after cherry-pick
rach-id May 22, 2025
349921d
chore: comment debug statements and necessary cached Tx changes
rach-id May 22, 2025
fb04cf7
chore: synchronise access to multiple resources
rach-id May 22, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/auto_request_review.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# More info at https://github.com/necojackarc/auto-request-review
reviewers:
defaults:
- evan-forbes
- tzdybal
- rach-id

options:
ignore_draft: true
ignored_keywords:
- DO NOT REVIEW
enable_group_assignment: false
12 changes: 6 additions & 6 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ jobs:
goos: ["linux"]
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.22"
- uses: actions/checkout@v4
go-version-file: "go.mod"
- uses: technote-space/get-diff-action@v6
with:
PATTERNS: |
Expand All @@ -41,10 +41,10 @@ jobs:
needs: build
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.22"
- uses: actions/checkout@v4
go-version-file: "go.mod"
- uses: technote-space/get-diff-action@v6
with:
PATTERNS: |
Expand All @@ -63,10 +63,10 @@ jobs:
needs: build
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.22"
- uses: actions/checkout@v4
go-version-file: "go.mod"
- uses: technote-space/get-diff-action@v6
with:
PATTERNS: |
Expand Down
11 changes: 5 additions & 6 deletions .github/workflows/check-generated.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ jobs:
check-mocks-metrics:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.22"
go-version-file: "go.mod"

- uses: actions/checkout@v4

- name: "Check generated mocks and metrics"
run: |
Expand All @@ -42,13 +42,12 @@ jobs:
check-proto:
runs-on: ubuntu-latest
steps:
- uses: actions/setup-go@v5
with:
go-version: "1.22"

- uses: actions/checkout@v4
with:
fetch-depth: 1 # we need a .git directory to run git diff
- uses: actions/setup-go@v5
with:
go-version-file: "go.mod"

- name: "Check protobuf generated code"
run: |
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/e2e-manual-multiversion.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.22'

- uses: actions/checkout@v4
go-version-file: 'go.mod'

- name: Build
working-directory: test/e2e
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/e2e-manual.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.22'

- uses: actions/checkout@v4
go-version-file: 'go.mod'

- name: Build
working-directory: test/e2e
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.22'
- uses: actions/checkout@v4
go-version-file: 'go.mod'
- uses: technote-space/get-diff-action@v6
with:
PATTERNS: |
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/fuzz-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,10 @@ jobs:
fuzz-nightly-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.22'

- uses: actions/checkout@v4
go-version-file: 'go.mod'

- name: Install go-fuzz
working-directory: test/fuzz
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/govulncheck.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ jobs:
govulncheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.23"
go-version-file: "go.mod"
check-latest: true
- uses: actions/checkout@v4
- uses: technote-space/get-diff-action@v6
with:
PATTERNS: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.22'
go-version-file: 'go.mod'
- uses: technote-space/get-diff-action@v6
with:
PATTERNS: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pre-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:

- uses: actions/setup-go@v5
with:
go-version: '1.22'
go-version-file: 'go.mod'

# Similar check to ./release-version.yml, but enforces this when pushing
# tags. The ./release-version.yml check can be bypassed and is mainly
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release-version.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:

- uses: actions/setup-go@v5
with:
go-version: '1.22'
go-version-file: 'go.mod'

- name: Check version
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

- uses: actions/setup-go@v5
with:
go-version: '1.22'
go-version-file: 'go.mod'

# Similar check to ./release-version.yml, but enforces this when pushing
# tags. The ./release-version.yml check can be bypassed and is mainly
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ jobs:
matrix:
part: ["00", "01", "02", "03", "04", "05"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.22"
- uses: actions/checkout@v4
go-version-file: "go.mod"
- uses: technote-space/get-diff-action@v6
with:
PATTERNS: |
Expand Down
3 changes: 3 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ linters-settings:
- google.golang.org/protobuf/types/known/timestamppb
- gonum.org/v1/gonum/stat
- github.com/celestiaorg/go-square/v2
- github.com/stretchr/testify/assert
- github.com/klauspost/reedsolomon
test:
files:
- "$test"
Expand All @@ -111,6 +113,7 @@ linters-settings:
- github.com/stretchr/testify
- github.com/decred/dcrd/dcrec/secp256k1/v4
- github.com/celestiaorg/go-square/v2
- github.com/stretchr/testify/require
revive:
enable-all-rules: true
rules:
Expand Down
3 changes: 2 additions & 1 deletion abci/types/mocks/application.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion blocksync/metrics.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
package blocksync

import (
"github.com/cometbft/cometbft/types"
"github.com/go-kit/kit/metrics"

"github.com/cometbft/cometbft/types"
)

const (
Expand Down
4 changes: 2 additions & 2 deletions blocksync/pool.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ eg, L = latency = 0.1s
const (
requestIntervalMS = 2
maxPendingRequestsPerPeer = 20
requestRetrySeconds = 30
requestRetrySeconds = 45

// Minimum recv rate to ensure we're receiving blocks from a peer fast
// enough. If a peer is not sending us data at at least that rate, we
Expand All @@ -54,7 +54,7 @@ const (
minBlocksForSingleRequest = 50
)

var peerTimeout = 15 * time.Second // not const so we can override with tests
var peerTimeout = 120 * time.Second // not const so we can override with tests

/*
Peers self report their heights when we join the block pool.
Expand Down
2 changes: 1 addition & 1 deletion blocksync/reactor.go
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ func (bcR *Reactor) GetChannels() []*p2p.ChannelDescriptor {
return []*p2p.ChannelDescriptor{
{
ID: BlocksyncChannel,
Priority: 5,
Priority: 30,
SendQueueCapacity: 1000,
RecvBufferCapacity: 50 * 4096,
RecvMessageCapacity: MaxMsgSize,
Expand Down
5 changes: 2 additions & 3 deletions blocksync/reactor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,9 @@ func newReactor(

lastExtCommit := seenExtCommit.Clone()

thisBlock := state.MakeBlock(blockHeight, types.MakeData([]types.Tx{}), lastExtCommit.ToCommit(), nil, state.Validators.Proposer.Address)

thisParts, err := thisBlock.MakePartSet(types.BlockPartSizeBytes)
thisBlock, thisParts, err := state.MakeBlock(blockHeight, types.MakeData([]types.Tx{}), lastExtCommit.ToCommit(), nil, state.Validators.Proposer.Address)
require.NoError(t, err)

blockID := types.BlockID{Hash: thisBlock.Hash(), PartSetHeader: thisParts.Header()}

// Simulate a commit for the current height
Expand Down
23 changes: 15 additions & 8 deletions consensus/byzantine_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ import (
"testing"
"time"

"github.com/cometbft/cometbft/consensus/propagation"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

Expand Down Expand Up @@ -57,11 +59,13 @@ func TestByzantinePrevoteEquivocation(t *testing.T) {
})
state, _ := stateStore.LoadFromDBOrGenesisDoc(genDoc)
thisConfig := ResetConfig(fmt.Sprintf("%s_%d", testName, i))
nodeKey, err := p2p.LoadOrGenNodeKey(thisConfig.NodeKeyFile())
require.NoError(t, err)
defer os.RemoveAll(thisConfig.RootDir)
ensureDir(path.Dir(thisConfig.Consensus.WalFile()), 0o700) // dir for wal
app := appFunc()
vals := types.TM2PB.ValidatorUpdates(state.Validators)
_, err := app.InitChain(context.Background(), &abci.RequestInitChain{Validators: vals})
_, err = app.InitChain(context.Background(), &abci.RequestInitChain{Validators: vals})
require.NoError(t, err)

blockDB := dbm.NewMemDB()
Expand Down Expand Up @@ -91,11 +95,14 @@ func TestByzantinePrevoteEquivocation(t *testing.T) {

// Make State
blockExec := sm.NewBlockExecutor(stateStore, log.TestingLogger(), proxyAppConnCon, mempool, evpool, blockStore)
cs := NewState(thisConfig.Consensus, state, blockExec, blockStore, mempool, evpool)
cs.SetLogger(cs.Logger)

// set private validator
pv := privVals[i]
propagationReactor := propagation.NewReactor(nodeKey.ID(), blockStore, mempool, pv, state.ChainID)
cs := NewState(thisConfig.Consensus, state, blockExec, blockStore, propagationReactor, mempool, evpool)
cs.SetLogger(cs.Logger)
cs.SetPrivValidator(pv)
propagationReactor.SetProposalVerifier(cs)

eventBus := types.NewEventBus()
eventBus.SetLogger(log.TestingLogger().With("module", "events"))
Expand All @@ -114,7 +121,7 @@ func TestByzantinePrevoteEquivocation(t *testing.T) {
blocksSubs := make([]types.Subscription, 0)
eventBuses := make([]*types.EventBus, nValidators)
for i := 0; i < nValidators; i++ {
reactors[i] = NewReactor(css[i], true) // so we dont start the consensus states
reactors[i] = NewReactor(css[i], css[i].propagator, true) // so we dont start the consensus states
reactors[i].SetLogger(css[i].Logger)

// eventBus is already started with the cs
Expand Down Expand Up @@ -210,7 +217,7 @@ func TestByzantinePrevoteEquivocation(t *testing.T) {
}
proposerAddr := lazyProposer.privValidatorPubKey.Address()

block, err := lazyProposer.blockExec.CreateProposalBlock(
block, _, err := lazyProposer.blockExec.CreateProposalBlock(
ctx, lazyProposer.Height, lazyProposer.state, extCommit, proposerAddr)
require.NoError(t, err)
blockParts, err := block.MakePartSet(types.BlockPartSizeBytes)
Expand Down Expand Up @@ -352,7 +359,7 @@ func TestByzantineConflictingProposalsWithPartition(t *testing.T) {
blocksSubs[i], err = eventBus.Subscribe(context.Background(), testSubscriber, types.EventQueryNewBlock)
require.NoError(t, err)

conR := NewReactor(css[i], true) // so we don't start the consensus states
conR := NewReactor(css[i], css[i].propagator, true) // so we don't start the consensus states
conR.SetLogger(logger.With("validator", i))
conR.SetEventBus(eventBus)

Expand Down Expand Up @@ -461,7 +468,7 @@ func byzantineDecideProposalFunc(ctx context.Context, t *testing.T, height int64
// Avoid sending on internalMsgQueue and running consensus state.

// Create a new proposal block from state/txs from the mempool.
block1, err := cs.createProposalBlock(ctx)
block1, _, err := cs.createProposalBlock(ctx)
require.NoError(t, err)
blockParts1, err := block1.MakePartSet(types.BlockPartSizeBytes)
require.NoError(t, err)
Expand All @@ -478,7 +485,7 @@ func byzantineDecideProposalFunc(ctx context.Context, t *testing.T, height int64
deliverTxsRange(t, cs, 0, 1)

// Create a new proposal block from state/txs from the mempool.
block2, err := cs.createProposalBlock(ctx)
block2, _, err := cs.createProposalBlock(ctx)
require.NoError(t, err)
blockParts2, err := block2.MakePartSet(types.BlockPartSizeBytes)
require.NoError(t, err)
Expand Down
Loading
Loading