Add eraE file format

README.md

@@ -5,7 +5,8 @@ Links to download [e2store](https://github.com/status-im/nimbus-eth2/blob/613f4a
 
 
 Historical data providers offer **three primary storage formats**:
-- [**Era1**(Pre-Merge Execution History)](./formats/era1.md) - Archive nodes providing **execution layer history** before The Merge (ETH1).
+- [**EraE**(Full Execution History)] - Archive nodes providing **execution layer history**.
+- [**Era1**(Pre-Merge Execution History)](./formats/era1.md) (*deprecated*) - Archive nodes providing **execution layer history** before The Merge (ETH1).
 - [**Era**(Beacon Chain History)](./formats/era.md) - Stores data from the genesis of the Beacon Chain onwards. Can be used by Execution layer clients for history **from The Merge onward**, including historical block data.
 - [**E2SS**(Execution State)](./formats/e2ss.md) - **State snapshots** for execution clients.
 - [**E2HS**(Execution Layer History)](./formats/e2hs.md) - **full execution layer history** for execution clients, provides data from genesis to latest, headers are accompanied by proofs of canonicalness.
@@ -20,7 +21,7 @@ No e2store type may be reused. A list of all defined E2store types can be found
 |---|---|---|---| 
 | [trin-e2store](https://github.com/ethereum/trin/tree/master/crates/e2store)  | Rust | [Era](./formats/era.md), [Era1](./formats/era1.md), [E2SS](./formats/e2ss.md), [E2HS](./formats/e2hs.md) |   | 
 | [@ethereumjs/e2store](https://github.com/ethereumjs/ethereumjs-monorepo/tree/master/packages/e2store) | Javascript | [Era](./formats/era.md), [Era1](./formats/era1.md), [E2HS](./formats/e2hs.md) |   |
-| [go-ethereum](https://github.com/ethereum/go-ethereum/tree/master/internal/era) | Go | [Era1](./formats/era1.md) |   |
+| [go-ethereum](https://github.com/ethereum/go-ethereum/tree/master/internal/era) | Go | [EraE](./formats/erae.md) |   |
 | [nimbus-eth1](https://github.com/status-im/nimbus-eth1/blob/master/portal/eth_data/era1.nim) | Nim | [Era1](./formats/era1.md) |   |
 | [nimbus-eth2](https://github.com/status-im/nimbus-eth2/blob/stable/ncli/era.nim) | Nim | [Era](./formats/era.md) |   |
 | [nimbus-e2store.py](https://github.com/status-im/nimbus-eth2/blob/stable/ncli/e2store.py) | Python | [Era](./formats/era.md) |   |

formats/era1.md

@@ -1,7 +1,7 @@
 # Era1 files
 
  Era1 files are themselves e2store files. For more information on this format,
- see https:github.com/status-im/nimbus-eth2/blob/stable/docs/e2store.md.
+ see https://github.com/status-im/nimbus-eth2/blob/stable/docs/e2store.md.
 
  The overall structure of an Era1 file follows closely the structure of an Era file
  which contains consensus Layer data (and as a byproduct, EL data after the merge).
@@ -37,4 +37,4 @@
  entries in the file is recorded in count.
 
  Due to the accumulator size limit of 8192, the maximum number of blocks in
- an Era1 batch is also 8192.
+ an Era1 batch is also 8192.

formats/erae.md

@@ -0,0 +1,59 @@
+# EraE files
+
+ EraE files are themselves e2store files. For more information on this format,
+ see https://github.com/status-im/nimbus-eth2/blob/stable/docs/e2store.md.
+
+The format is designed to be compatible for both pre-merge execution layer data
+and post-merge. It is intended to be a companion to the Era format.
+
+## Specification
+
+The format can be summarized with the following expression:
+
+       eraE := Version | CompressedHeader+ | CompressedBody+ | CompressedReceipts+ | Proofs+ | TotalDifficulty* | other-entries* | Accumulator? | BlockIndex
+
+Each basic element is its own e2store entry:
+
+       Version            = { type: 0x3265, data: nil }
+       CompressedHeader   = { type: 0x03,   data: snappyFramed(rlp(header)) }
+       CompressedBody     = { type: 0x04,   data: snappyFramed(rlp(body)) }
+       CompressedSlimReceipts = { type: 0x08,   data: snappyFramed(rlp([tx-type, post-state-or-status, cumulative-gas, logs])) }
+       TotalDifficulty    = { type: 0x06,   data: uint256(header.total_difficulty) }
+       Proof              = { type: 0x09    data: snappyFramed(rlp([proof-type, ssz(BlockProofHistoricalHashesAccumulator) | ssz(BlockProofHistoricalRoots) | ssz(BlockProofHistoricalSummaries)]))}
+       AccumulatorRoot    = { type: 0x07,   data: hash_tree_root(List(HeaderRecord, 8192)) }
+       Index              = { type: 0x3267, data: index }
+
+A few notes on individual elements:
+
+- `CompressedReceipts` is optional and a different format than the consensus EIP-2718 format to optimize internal use in clients.
+- `Proof` is optional but if included, provides the coresponding proof for each `CompressedHeader` in the file corresponding to the Portal Network proofs specification[^1]. It's possible to have multiple proof types in the same file at fork boundaries.
+- `TotalDifficulty` is optional and little-endian encoded.
+- `AccumulatorRoot` is optional and only defined for epochs with pre-merge data.
+- `HeaderRecord` is defined in the Portal Network specification[^2].
+- `other-entries` is a placeholder for other potential objects to be added to EraE.
+
+`BlockIndex` stores relative offsets to the components of each block entry. This allows `O(1)` access to all components of a block. The format is:
+
+       block-index := starting-number | indexes | indexes | indexes ... | component-count | count
+
+Where `indexes` represents the index of each block component:
+
+       indexes := header-index | body-index | receipts-index | difficulty-index? | proof-index?
+
+The value `component-count` is the number of indexes stored by `indexes`. It should be in the range of 3-5, depending on whether `TotalyDifficulty` and `Proofs` are present 
+
+All values in the block index are little-endian `uint64`.
+
+`starting-number` is the first block number in the archive. Every index is a defined relative to index's location in the file. The total number of block entries in the file is recorded in count.
+
+Due to the accumulator size limit of 8192, the maximum number of blocks in an Era batch is also 8192. This is also the value of `SLOTS_PER_HISTORICAL_ROOT`[^3] on the Beacon chain, so it is nice to align on the value.
+
+### Merge transition
+
+There are some small differences between pre-merge and post-merge `eraE` files:
+- `TotalDifficulty` should only be encoded for `eraE` files pre-merge. For the epoch where the merge occurs, fill all remaining post-merge blocks with the final total difficulty of the chain.
+- `AccumulatorRoot` should only be encoded for `eraE` files pre-merge. For the epoch where the merge occurs, compute the root for an incomplete epoch where only pre-merge blocks are recorded in the accumulator. 
+
+[^1]: https://github.com/ethereum/portal-network-specs/blob/master/history/history-network.md#block-header
+[^2]: https://github.com/ethereum/portal-network-specs/blob/master/history/history-network.md#the-historical-hashes-accumulator
+[^3]: https://github.com/ethereum/consensus-specs/blob/44ae6e661d9beac383f4a1f33be74259bae93c85/presets/mainnet/phase0.yaml#L42