Validator produces block before chain is fully synced

[anshalshukla]

an-02 15:49:18.362 [info] (zeam): [node] peer connected: 16Uiu2HAmPQhkD6Zg5Co2ee8ShshkiY4tDePKFARPpCS2oKSLj1E1(ream_0), total peers: 2
Jan-02 15:49:18.363 [info] (zeam): [network] rust-bridge: [reqresp] Sent StatusV1 request to 16Uiu2HAmPQhkD6Zg5Co2ee8ShshkiY4tDePKFARPpCS2oKSLj1E1 (id: 2)
Jan-02 15:49:18.363 [info] (zeam): [node] sent status request to peer 16Uiu2HAmPQhkD6Zg5Co2ee8ShshkiY4tDePKFARPpCS2oKSLj1E1(ream_0): request_id=2, head_slot=0, finalized_slot=0
Jan-02 15:49:18.962 [info] (zeam): [chain] 
+===============================================================+
  CHAIN STATUS: Current Slot: 563 | Head Slot: 0 | Behind: 563
+---------------------------------------------------------------+
  Connected Peers:    2
+---------------------------------------------------------------+
  Head Block Root:    0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b
  Parent Block Root:  0x0000000000000000000000000000000000000000000000000000000000000000
  State Root:         0xbcfbd10b2dbf05b8700cd5ab4d54923778ae5f3be13269fb9a70e41a690812dc
  Timely:             YES
+---------------------------------------------------------------+
  Latest Justified:   Slot      0 | Root: 0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b
  Latest Finalized:   Slot      0 | Root: 0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b
+===============================================================+

Jan-02 15:49:18.962 [info] (zeam): [validator] constructing attestation message for slot=563
Jan-02 15:49:18.962 [info] (zeam): [chain] constructing attestation data at slot=563 with chain head={"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0} safe_target={"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0}
Jan-02 15:49:18.962 [info] (zeam): [chain] calculated target for attestations at slot=563: {"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0}
Jan-02 15:49:18.971 [info] (zeam): [validator] constructed attestation slot=563 validator=0
Jan-02 15:49:18.971 [info] (zeam): [node] adding locally produced attestation to chain: slot=563 validator=0
Jan-02 15:49:18.973 [info] (zeam): [node] published attestation to network: slot=563 validator=0
Jan-02 15:49:19.005 [info] (zeam): [node] received gossip attestation for slot=563 validator=2 from peer=unknown_peer
Jan-02 15:49:19.005 [warning] (zeam): [chain] gossip attestation validation failed: error.UnknownSourceBlock
Jan-02 15:49:19.013 [info] (zeam): [network] rust-bridge: Connection established with peer: 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv
Jan-02 15:49:19.013 [info] (zeam): [network] network-0:: Peer connected: 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv(lantern_0)
Jan-02 15:49:19.013 [info] (zeam): [node] peer connected: 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv(lantern_0), total peers: 3
Jan-02 15:49:19.013 [info] (zeam): [network] rust-bridge: [reqresp] Sent StatusV1 request to 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv (id: 3)
Jan-02 15:49:19.013 [info] (zeam): [node] sent status request to peer 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv(lantern_0): request_id=3, head_slot=0, finalized_slot=0
Jan-02 15:49:19.020 [info] (zeam): [network] rust-bridge: [reqresp] Received response from 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv for request id 3 (103 bytes)
Jan-02 15:49:19.020 [info] (zeam): [node] received status response from peer 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv(lantern_0) head_slot=563, finalized_slot=543
Jan-02 15:49:19.021 [info] (zeam): [node] received gossip attestation for slot=563 validator=1 from peer=unknown_peer
Jan-02 15:49:19.021 [warning] (zeam): [chain] gossip attestation validation failed: error.UnknownSourceBlock
Jan-02 15:49:19.032 [info] (zeam): [network] rust-bridge: [reqresp] Received request from 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv for protocol /leanconsensus/req/status/1/ssz_snappy (102 bytes)
Jan-02 15:49:19.032 [info] (zeam): [network] rust-bridge: [reqresp] Sent response payload on channel 1 (peer: 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv)
Jan-02 15:49:19.032 [info] (zeam): [network] rust-bridge: [reqresp] Sent end-of-stream on channel 1 (peer: 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv)
Jan-02 15:49:19.058 [info] (zeam): [node] received gossip attestation for slot=563 validator=3 from peer=unknown_peer
Jan-02 15:49:19.058 [warning] (zeam): [chain] gossip attestation validation failed: error.UnknownSourceBlock
Jan-02 15:49:22.005 [info] (zeam): [validator] produced block for slot=564 proposer=0 with root=3cc3f11188b2f34b79df4857f54e34507f2e4ac6f1fa7e20a11c50cf565eb2a9
Jan-02 15:49:22.005 [info] (zeam): [chain] constructing attestation data at slot=564 with chain head={"root":"0x3cc3f11188b2f34b79df4857f54e34507f2e4ac6f1fa7e20a11c50cf565eb2a9","slot":564} safe_target={"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0}
Jan-02 15:49:22.005 [info] (zeam): [chain] calculated target for attestations at slot=564: {"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0}
Jan-02 15:49:22.005 [info] (zeam): [validator] packing proposer attestation for slot=564 proposer=0: {"slot":564,"head":{"root":"0x3cc3f11188b2f34b79df4857f54e34507f2e4ac6f1fa7e20a11c50cf565eb2a9","slot":564},"target":{"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0},"source":{"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0}}
Jan-02 15:49:22.008 [info] (zeam): [validator] signed produced block with attestation for slot=564 root=3cc3f11188b2f34b79df4857f54e34507f2e4ac6f1fa7e20a11c50cf565eb2a9
Jan-02 15:49:22.008 [info] (zeam): [node] adding produced signed block to the chain: slot=564 proposer=0
Jan-02 15:49:22.010 [info] (zeam): [chain] processed block with root=0x3cc3f11188b2f34b79df4857f54e34507f2e4ac6f1fa7e20a11c50cf565eb2a9 slot=564 processing time=0.000007373 (computed root=false computed state=false)
Jan-02 15:49:22.011 [info] (zeam): [node] published block to network: slot=564 proposer=0
Jan-02 15:49:22.011 [info] (zeam): [chain] completed on block followup with states_count=2 fc_nodes_count=2
Jan-02 15:49:23.000 [info] (zeam): [chain] 
+===============================================================+
  CHAIN STATUS: Current Slot: 564 | Head Slot: 564 | Behind: 0
+---------------------------------------------------------------+
  Connected Peers:    3
+---------------------------------------------------------------+
  Head Block Root:    0x3cc3f11188b2f34b79df4857f54e34507f2e4ac6f1fa7e20a11c50cf565eb2a9
  Parent Block Root:  0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b
  State Root:         0x7459c887c78d843f8059090371b7c985bbcade12fb2a5ef5026aa20014bf7122
  Timely:             YES
+---------------------------------------------------------------+
  Latest Justified:   Slot      0 | Root: 0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b
  Latest Finalized:   Slot      0 | Root: 0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b
+===============================================================+

Jan-02 15:49:23.000 [info] (zeam): [validator] constructing attestation message for slot=564
Jan-02 15:49:23.001 [info] (zeam): [chain] constructing attestation data at slot=564 with chain head={"root":"0x3cc3f11188b2f34b79df4857f54e34507f2e4ac6f1fa7e20a11c50cf565eb2a9","slot":564} safe_target={"root":"0xe88283fbcf6d34682f727737f47f5f86a7c7d153d44ed3cb0c0cdb511b17db4b","slot":0}

[g11tech]

since there are no slashings, I am not sure if we need to solve this problem as eventually the branch with the most network behind it should win by forkchoice rules once they are synced back

only thing to consider might be to not mint blocks behind what could be the "networkwide" finalized, which one might need to see whats the max finalized amongst all peers