Geth 1.16.0 + Lighthouse 7.0.0 private PoS chain: Geth never becomes synced, Lighthouse reports "Not ready Bellatrix"

[Light-k]

Body:

I am trying to set up a private PoS chain using Geth 1.16.0 and Lighthouse 7.0.0. During deployment, I noticed that Geth never enters a "synced" state, regardless of whether Lighthouse is running or not.

Geth log (tail):

text
INFO [05-29|08:16:22.564] IPC endpoint opened url=/root/blockChain/data-node1/geth.ipc
INFO [05-29|08:16:22.564] Loaded JWT secret file path=jwt.hex crc32=0xa721f46c
INFO [05-29|08:16:22.564] New local node record seq=1,780,039,737,675 id=8a63fae02902eb09 ip=127.0.0.1 udp=0 tcp=30303
INFO [05-29|08:16:22.564] Started P2P networking self="enode://4ae86b7bad57a17003842ccefda9d31e38404b2871c05057c85d5c06df44b462dc963c3b5caea0778e03cceeb17a28ebc77775c0358ff20bec32c8ab0a335942@127.0.0.1:30303?discport=0"
INFO [05-29|08:16:22.564] HTTP server started endpoint=127.0.0.1:8545 auth=false prefix= cors= vhosts=localhost
INFO [05-29|08:16:22.564] WebSocket enabled url=ws://127.0.0.1:8551
INFO [05-29|08:16:22.564] HTTP server started endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost
INFO [05-29|08:16:22.564] Started log indexer
INFO [05-29|08:16:22.572] NAT mapped port proto=TCP extport=30303 intport=30303 interface=NAT-PMP(192.168.0.1)
WARN [05-29|08:16:57.400] Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!
Lighthouse beacon node log (tail):

text
May 29 08:12:45.001 WARN Not ready Bellatrix info: The execution endpoint is connected and configured, however it is not yet synced, service: slot_notifier
May 29 08:12:57.000 WARN Low peer count peer_count: 0, service: slot_notifier
May 29 08:12:57.001 INFO Searching for peers current_slot: 167, head_slot: 0, finalized_epoch: 0, finalized_root: 0x86e2a9795eb081602ea4a4b408d00e5868231c0f0011bb83827f235939e4d5ff, peers: 0, service: slot_notifier
May 29 08:12:57.001 WARN Not ready Bellatrix info: The execution endpoint is connected and configured, however it is not yet synced, service: slot_notifier
May 29 08:13:09.001 WARN Low peer count peer_count: 0, service: slot_notifier
May 29 08:13:09.001 INFO Searching for peers current_slot: 168, head_slot: 0, finalized_epoch: 0, finalized_root: 0x86e2a9795eb081602ea4a4b408d00e5868231c0f0011bb83827f235939e4d5ff, peers: 0, service: slot_notifier
May 29 08:13:09.001 WARN Not ready Bellatrix info: The execution endpoint is connected and configured, however it is not yet synced, service: slot_notifier
May 29 08:13:21.001 WARN Low peer count peer_count: 0, service: slot_notifier
May 29 08:13:21.001 INFO Searching for peers current_slot: 169, head_slot: 0, finalized_epoch: 0, finalized_root: 0x86e2a9795eb081602ea4a4b408d00e5868231c0f0011bb83827f235939e4d5ff, peers: 0, service: slot_notifier
May 29 08:13:21.001 WARN Not ready Bellatrix info: The execution endpoint is connected and configured, however it is not yet synced, service: slot_notifier
What I have tried and verified:

Confirmed that the same JWT secret is used by both Geth and Lighthouse (--authrpc.jwtsecret and --execution-jwt point to identical jwt.hex file).

Confirmed that Geth is running on the correct private network (chainId = 2358 via eth_chainId RPC call, and genesis.json is properly loaded).

Waited for a long time (slots keep advancing but head_slot stays 0).

Manually invoked engine_exchangeTransitionConfigurationV1 and engine_forkchoiceUpdatedV2 – both returned VALID and temporarily resolved the issue, but the problem reappears after restart.

My genesis.json configuration (placeholders ${ADDR1}, ${ADDR2}, ${ADDR3} for actual addresses):

json
{
"config": {
"chainId": 2358,
"homesteadBlock": 0,
"eip150Block": 0,
"eip155Block": 0,
"eip158Block": 0,
"byzantiumBlock": 0,
"constantinopleBlock": 0,
"petersburgBlock": 0,
"istanbulBlock": 0,
"berlinBlock": 0,
"londonBlock": 0,
"mergeNetsplitBlock": 0,
"terminalTotalDifficulty": 0,
"terminalTotalDifficultyPassed": true,
"shanghaiTime": 0,
"cancunTime": 0,
"blobSchedule": {
"cancun": {
"target": 3,
"max": 6,
"baseFeeUpdateFraction": 3338477
},
"prague": {
"target": 6,
"max": 9,
"baseFeeUpdateFraction": 5007716
}
}
},
"difficulty": "0",
"gasLimit": "30000000",
"alloc": {
"0x${ADDR1}": {"balance": "100000000000000000000"},
"0x${ADDR2}": {"balance": "100000000000000000000"},
"0x${ADDR3}": {"balance": "100000000000000000000"}
}
}
My Lighthouse config.yaml and related testnet files:

consensus/config/config.yaml:

yaml
PRESET_BASE: 'mainnet'
CONFIG_NAME: 'my-pos-chain'

创世配置

MIN_GENESIS_ACTIVE_VALIDATOR_COUNT: 0
MIN_GENESIS_TIME: 0
GENESIS_FORK_VERSION: 0x00000000
GENESIS_DELAY: 0

时间参数

SECONDS_PER_SLOT: 12
SLOTS_PER_EPOCH: 32
SECONDS_PER_ETH1_BLOCK: 14
ETH1_FOLLOW_DISTANCE: 1
EPOCHS_PER_ETH1_VOTING_PERIOD: 4
SLOTS_PER_HISTORICAL_ROOT: 8192
MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 256
SHARD_COMMITTEE_PERIOD: 256
MAX_EPOCHS_PER_CROSSLINK: 64
MIN_ATTESTATION_INCLUSION_DELAY: 1

验证者惩罚参数

INACTIVITY_SCORE_BIAS: 4
INACTIVITY_SCORE_RECOVERY_RATE: 16
EJECTION_BALANCE: 16000000000
MIN_PER_EPOCH_CHURN_LIMIT: 4
CHURN_LIMIT_QUOTIENT: 65536

Merge / TTD 配置

TERMINAL_TOTAL_DIFFICULTY: 0
TERMINAL_BLOCK_HASH: 0x0000000000000000000000000000000000000000000000000000000000000000
TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH: 18446744073709551615

分叉版本号（各分叉从 Epoch 0 开始）

ALTAIR_FORK_EPOCH: 0
ALTAIR_FORK_VERSION: 0x00000001
BELLATRIX_FORK_EPOCH: 0
BELLATRIX_FORK_VERSION: 0x00000002
CAPELLA_FORK_EPOCH: 0
CAPELLA_FORK_VERSION: 0x00000003
DENEB_FORK_EPOCH: 0
DENEB_FORK_VERSION: 0x00000004

Deposit 合约

DEPOSIT_CHAIN_ID: 2358
DEPOSIT_NETWORK_ID: 2358
DEPOSIT_CONTRACT_ADDRESS: 0x0000000000000000000000000000000000000000
consensus/config/deploy_block.txt: 0

consensus/config/deposit_contract_block.txt: 0

consensus/config/deposit_contract.txt: 0x0000000000000000000000000000000000000000

Command to generate the genesis state (using eth-genesis-state-generator):

bash
eth-genesis-state-generator beaconchain
--eth1-config genesis.json
--config ./consensus/config/config.yaml
--mnemonics ./consensus/config/mnemonics.yaml
--json-output ./consensus/config/genesis.json
--state-output ./consensus/config/genesis.ssz
Geth startup command:

bash
nohup geth
--datadir ./data-node1
--networkid 2358
--port 30303
--http --http.addr 127.0.0.1 --http.port 8545 --http.api "eth,net,web3,engine,admin" --http.corsdomain ""
--ws --ws.addr 127.0.0.1 --ws.port 8546
--authrpc.addr 127.0.0.1 --authrpc.port 8551 --authrpc.jwtsecret ./jwt.hex --authrpc.vhosts ""
--allow-insecure-unlock
--maxpeers 25
--syncmode full
2>&1 > geth1.log &
Lighthouse beacon node startup command:

bash
nohup lighthouse bn
--testnet-dir ./consensus/config
--datadir ./consensus/bn1
--execution-endpoint http://127.0.0.1:8551 --execution-jwt ./jwt.hex --execution-timeout-multiplier 100
--genesis-state-url "file://$(pwd)/consensus/config/genesis.ssz"
--allow-insecure-genesis-sync
--disable-deposit-contract-sync
--disable-peer-scoring
--http --http-address 127.0.0.1 --http-port 5052
--port 9000
2>&1 > lighthouse_bn1.log &
Additional observation:

Even when only Geth is started (without Lighthouse), the eth_syncing endpoint always returns the following object instead of false:

json
{"jsonrpc":"2.0","id":1,"result":{"currentBlock":"0x0","healedBytecodeBytes":"0x0","healedBytecodes":"0x0","healedTrienodeBytes":"0x0","healedTrienodes":"0x0","healingBytecode":"0x0","healingTrienodes":"0x0","highestBlock":"0x0","startingBlock":"0x0","stateIndexRemaining":"0x0","syncedAccountBytes":"0x0","syncedAccounts":"0x0","syncedBytecodeBytes":"0x0","syncedBytecodes":"0x0","syncedStorage":"0x0","syncedStorageBytes":"0x0","txIndexFinishedBlocks":"0x0","txIndexRemainingBlocks":"0x1"}}
This indicates Geth is stuck in a "not synced" state even on a fresh private chain. The field txIndexRemainingBlocks: 0x1 suggests an unfinished indexing task that never completes. I expect eth_syncing to return false as this is a new chain with no blocks to sync.

Request:
Could the official team or anyone from the community help me debug this? Ideally, I would really appreciate remote assistance (e.g., via SSH) because I have been stuck on this for a very long time.

Thank you!

[ozpool]

The geth side here is actually fine, it's idle on purpose. Geth only moves forward when the consensus client drives it over the Engine API (forkchoiceUpdated / newPayload). That no consensus updates received in a while line is the else-branch in eth/catalyst/api.go - geth got the initial handshake from Lighthouse but then stopped getting updates, because Lighthouse itself isn't following a chain yet.

The Lighthouse logs are where the real cause is: head_slot: 0, peers: 0, Not ready Bellatrix ... not yet synced. The beacon chain never produces a block past genesis, so there's nothing to hand to geth, so geth never leaves genesis. "Geth never becomes synced" is the expected symptom of that - it's a CL/config thing, not a geth bug.

For a private chain that's meant to be merged from genesis, I'd check these:

1. EL genesis - terminalTotalDifficulty: 0 and terminalTotalDifficultyPassed: true in the geth genesis.json config block.
2. CL config (config.yaml) - TERMINAL_TOTAL_DIFFICULTY: 0, plus the fork epochs set at genesis: ALTAIR_FORK_EPOCH: 0, BELLATRIX_FORK_EPOCH: 0 (and Capella/Deneb/etc for whatever version you're on). "Not ready Bellatrix" almost always means Bellatrix hasn't activated in the CL config.
3. A validator client actually running with active genesis validators. With head_slot: 0 and nothing being proposed, a beacon node with no validator will just sit at slot 0 forever, even though the engine handshake looks healthy.

Honestly the fastest way to get a working setup is to start from a known-good generator instead of hand-rolling genesis - Kurtosis ethereum-package or eth-pos-devnet both wire up EL+CL+validator with matching genesis and fork epochs, and you can diff your config against theirs.

This is more of a deployment/config question than a geth bug though, so you'll probably get a faster back-and-forth on the Geth discord (#node-operators) than here.