diff --git a/README.md b/README.md index 0bc7974..0cfa511 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,8 @@ Links to download [e2store](https://github.com/status-im/nimbus-eth2/blob/613f4a Historical data providers offer **three primary storage formats**: -- [**Era1**(Pre-Merge Execution History)](./formats/era1.md) - Archive nodes providing **execution layer history** before The Merge (ETH1). +- [**EraE**(Full Execution History)] - Archive nodes providing **execution layer history**. +- [**Era1**(Pre-Merge Execution History)](./formats/era1.md) (*deprecated*) - Archive nodes providing **execution layer history** before The Merge (ETH1). - [**Era**(Beacon Chain History)](./formats/era.md) - Stores data from the genesis of the Beacon Chain onwards. Can be used by Execution layer clients for history **from The Merge onward**, including historical block data. - [**E2SS**(Execution State)](./formats/e2ss.md) - **State snapshots** for execution clients. - [**E2HS**(Execution Layer History)](./formats/e2hs.md) - **full execution layer history** for execution clients, provides data from genesis to latest, headers are accompanied by proofs of canonicalness. @@ -20,7 +21,7 @@ No e2store type may be reused. A list of all defined E2store types can be found |---|---|---|---| | [trin-e2store](https://github.com/ethereum/trin/tree/master/crates/e2store) | Rust | [Era](./formats/era.md), [Era1](./formats/era1.md), [E2SS](./formats/e2ss.md), [E2HS](./formats/e2hs.md) | | | [@ethereumjs/e2store](https://github.com/ethereumjs/ethereumjs-monorepo/tree/master/packages/e2store) | Javascript | [Era](./formats/era.md), [Era1](./formats/era1.md), [E2HS](./formats/e2hs.md) | | -| [go-ethereum](https://github.com/ethereum/go-ethereum/tree/master/internal/era) | Go | [Era1](./formats/era1.md) | | +| [go-ethereum](https://github.com/ethereum/go-ethereum/tree/master/internal/era) | Go | [EraE](./formats/erae.md) | | | [nimbus-eth1](https://github.com/status-im/nimbus-eth1/blob/master/portal/eth_data/era1.nim) | Nim | [Era1](./formats/era1.md) | | | [nimbus-eth2](https://github.com/status-im/nimbus-eth2/blob/stable/ncli/era.nim) | Nim | [Era](./formats/era.md) | | | [nimbus-e2store.py](https://github.com/status-im/nimbus-eth2/blob/stable/ncli/e2store.py) | Python | [Era](./formats/era.md) | | diff --git a/formats/era1.md b/formats/era1.md index d6d0ff1..cc20916 100644 --- a/formats/era1.md +++ b/formats/era1.md @@ -1,7 +1,7 @@ # Era1 files Era1 files are themselves e2store files. For more information on this format, - see https:github.com/status-im/nimbus-eth2/blob/stable/docs/e2store.md. + see https://github.com/status-im/nimbus-eth2/blob/stable/docs/e2store.md. The overall structure of an Era1 file follows closely the structure of an Era file which contains consensus Layer data (and as a byproduct, EL data after the merge). @@ -37,4 +37,4 @@ entries in the file is recorded in count. Due to the accumulator size limit of 8192, the maximum number of blocks in - an Era1 batch is also 8192. \ No newline at end of file + an Era1 batch is also 8192. diff --git a/formats/ere.md b/formats/ere.md new file mode 100644 index 0000000..0daec82 --- /dev/null +++ b/formats/ere.md @@ -0,0 +1,70 @@ +# Ere files + + Era execution (ere) files are themselves e2store files. For more information on this format, + see https://github.com/status-im/nimbus-eth2/blob/stable/docs/e2store.md. + +The format is designed to be compatible for both pre-merge execution layer data +and post-merge. It is intended to be a companion to the Era format. + +## Specification + +The format can be summarized with the following expression: + + ere := Version | CompressedHeader+ | CompressedBody+ | CompressedSlimReceipts+ | Proofs+ | TotalDifficulty* | other-entries* | Accumulator? | BlockIndex + +Each basic element is its own e2store entry: + + Version = { type: [0x65, 0x32], data: nil } + CompressedHeader = { type: [0x03, 0x00], data: snappyFramed(rlp(header)) } + CompressedBody = { type: [0x04, 0x00], data: snappyFramed(rlp(body)) } + CompressedSlimReceipts = { type: [0x0a, 0x00], data: snappyFramed(rlp([tx-type, post-state-or-status, cumulative-gas, logs])) } + TotalDifficulty = { type: [0x06, 0x00], data: uint256(header.total_difficulty) } + Proof = { type: [0x0b, 0x00], data: snappyFramed(rlp([proof-type, ssz(BlockProofHistoricalHashesAccumulator) | ssz(BlockProofHistoricalRoots) | ssz(BlockProofHistoricalSummariesCapella) | ssz(BlockProofHistoricalSummariesDeneb)]))} + AccumulatorRoot = { type: [0x07, 0x00], data: hash_tree_root(List(HeaderRecord, 8192)) } + Index = { type: [0x66, 0x32], data: index } + +A few notes on individual elements: + +- `CompressedSlimReceipts` is optional and a different format than the consensus EIP-2718 format to optimize internal use in clients. +- `Proof` is a *highly recommended* component that provides the corresponding proof for each `CompressedHeader` in the file corresponding to the Portal Network proofs specification[^1]. It's possible to have multiple proof types in the same file at fork boundaries. +- `TotalDifficulty` is optional and little-endian encoded. +- `AccumulatorRoot` is optional and only defined for epochs with pre-merge data. +- `HeaderRecord` is defined in the Portal Network specification[^2]. +- `other-entries` is a placeholder for other potential objects to be added to Ere. + +`BlockIndex` stores relative offsets to the components of each block entry. This allows `O(1)` access to all components of a block. The format is: + + block-index := starting-number | indexes | indexes | indexes ... | component-count | count + +Where `indexes` represents the index of each block component: + + indexes := header-index | body-index | receipts-index | difficulty-index? | proof-index? + +The value `component-count` is the number of indexes stored by `indexes`. It should be in the range of 2-5, depending on whether `CompressedSlimReceipts`, `TotalDifficulty` and `Proofs` are present. + +All values in the block index are little-endian `uint64`. + +`starting-number` is the first block number in the archive. Every index is a defined relative to index's location in the file. The total number of block entries in the file is recorded in count. + +Due to the accumulator size limit of 8192, the maximum number of blocks in an Era batch is also 8192. This is also the value of `SLOTS_PER_HISTORICAL_ROOT`[^3] on the Beacon chain, so it is nice to align on the value. + +### Merge transition + +There are some small differences between pre-merge and post-merge `ere` files: +- `TotalDifficulty` should only be encoded for `ere` files pre-merge. For the epoch where the merge occurs, fill all remaining post-merge blocks with the final total difficulty of the chain. +- `AccumulatorRoot` should only be encoded for `ere` files pre-merge. For the epoch where the merge occurs, compute the root for an incomplete epoch where only pre-merge blocks are recorded in the accumulator. + +### Proof type + +The `proof-type` value maps to an associated proof object. It's used to disambiguate the raw SSZ data inside RLP objects. + +| Type | Proof object | +|---|---| +| 0 | BlockProofHistoricalHashesAccumulator | +| 1 | BlockProofHistoricalRoots | +| 2 | BlockProofHistoricalSummariesCapella | +| 3 | BlockProofHistoricalSummariesDeneb | + +[^1]: https://github.com/ethereum/portal-network-specs/blob/master/legacy/history/history-network.md#block-header +[^2]: https://github.com/ethereum/portal-network-specs/blob/master/legacy/history/history-network.md#the-historical-hashes-accumulator +[^3]: https://github.com/ethereum/consensus-specs/blob/44ae6e661d9beac383f4a1f33be74259bae93c85/presets/mainnet/phase0.yaml#L42 diff --git a/types/0x0a00.md b/types/0x0a00.md new file mode 100644 index 0000000..9ea319b --- /dev/null +++ b/types/0x0a00.md @@ -0,0 +1,8 @@ +# CompressedSlimReceipts + +``` +type: [0x0a, 0x00] +data: snappyFramed(rlp(receipts)) +``` + +`CompressedSlimReceipts` entries contain snappy compressed rlp execution receipts, without bloom filters. diff --git a/types/0x0b00.md b/types/0x0b00.md new file mode 100644 index 0000000..7c00d9e --- /dev/null +++ b/types/0x0b00.md @@ -0,0 +1,9 @@ +# Proof + +``` +type: [0x0b, 0x00] +data: snappyFramed(rlp([proof-type, ssz(BlockProofHistoricalHashesAccumulator) | ssz(BlockProofHistoricalRoots) | ssz(BlockProofHistoricalSummaries)])) +``` + +`Proof` entries contain a compressed RLP object with a proof type indicator and +specific proof format that can attest to the validity of the EraE file. diff --git a/types/README.md b/types/README.md index dad53fc..ad8cbfa 100644 --- a/types/README.md +++ b/types/README.md @@ -14,6 +14,8 @@ This is a sorted list by type number | [0x0700](0x0700.md) | Accumulator | [Era1](../formats/era1.md) | | | [0x0800](0x0800.md) | CompressedAccount | [E2SS](../formats/e2ss.md) | | | [0x0900](0x0900.md) | CompressedStorage | [E2SS](../formats/e2ss.md) | | +| [0x0a00](0x0a00.md) | CompressedSlimReceipts | [EraE](../formats/erae.md) | | +| [0x0b00](0x0b00.md) | Proof | [EraE](../formats/erae.md) | | | [0x6532](0x6532.md) | Version | ALL | | | [0x6632](0x6632.md) | BlockIndex | [Era1](../formats/era1.md), [E2HS](../formats/e2hs.ms) | | | [0x6932](0x6932.md) | SlotIndex | [Era](../formats/era.md) | |