Skip to content

Update EIP-7745: Fix some typos #9496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions EIPS/eip-7745.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Bloom filters are only useful as long as they are sufficiently sparse. False pos
- _log value index_: values are globally mapped to a linear index space, with a monotonically increasing _log value index_ assigned to each added _log value_. The _log values_ are added in the order of EVM execution (_address value_ first, then the _topic values_) so the logs generated in each block and in each transaction of the block occupy a continuous range in the index space. A _block delimiter_ is also added between blocks which has its own _log value index_ and is added to the Merkle tree of _log entries_ but not to the _filter maps_.
- _log entry_: an SSZ encoded log event with position metadata (a `LogEntry` container) is added to the `log_entries` Merkle tree at the first _log value index_ assigned to the event (the one assigned to the _address value_). The entries at indices assigned to _topic values_ are left empty (a `LogEntry` with all zero fields). The position metadata contains the block number but not the block hash since the block hash is not known yet when the block where the newly added logs were emitted is still being constructed. If needed, the block hash can be found in the _block delimiter_ positioned after the logs emitted in the given block (except for the head block which does not have a _block delimiter_ yet but its hash is always expected to be known to the prover/verifier).
- _block delimiter_: a special entry in the _log value index_ space that is placed between the logs emitted by each block. In order to fit into the fixed shape tree structure its Merkle tree shape is identical to the `LogEntry` container, with the `Log` part being empty and the meta info part is a `BlockDelimiterMeta` container instead of a `LogMeta`. The two are always distinguishable based on the `dummy_value` field which goes in place of the `log_in_tx_index` of log entries and always has a value of 2**64-1. It references the block by number and hash. Each _block delimiter_ is placed after the logs emitted by the referenced block when the next block is added (when the hash of the referenced block is already known), before the logs emitted by that block. This makes it easy to prove the _log value index_ boundaries of a searched block range and also allows proving the hash of the blocks emitting the matching logs which is required for filling out the position metainfo of the RPC response.
- _filter map_: a `MAP_WIDTH` by `MAP_HEIGHT` sized sparse bit map intended to help searching for _log values_ in a fixed `VALUES_PER_MAP` length section of the _log value index_ space. Each _log value_ is marked on the map at a row and column that depends on the _log value index_ and the _log value_ itself. Rows are sparsely encoded as a list of marked column indices (in strictly ascending order, which also coincides with the order of occurence). Each map contains at most `VALUES_PER_MAP` marks and therefore the chance of false positives is kept at a constant low level.
- _filter map_: a `MAP_WIDTH` by `MAP_HEIGHT` sized sparse bit map intended to help searching for _log values_ in a fixed `VALUES_PER_MAP` length section of the _log value index_ space. Each _log value_ is marked on the map at a row and column that depends on the _log value index_ and the _log value_ itself. Rows are sparsely encoded as a list of marked column indices (in strictly ascending order, which also coincides with the order of occurrence). Each map contains at most `VALUES_PER_MAP` marks and therefore the chance of false positives is kept at a constant low level.
- _filter epoch_: a `MAPS_PER_EPOCH` sized group of consecutive _filter maps_ stored in the hash tree in a way so that multiple rows of adjacent _filter maps_ with the same _row index_ can be efficiently retrieved in a single Merkle multiproof. The _log value_ to _row index_ mapping is constant during a single epoch but changes between epochs.

### Consensus data format
Expand Down Expand Up @@ -328,7 +328,7 @@ def is_potential_match(map_index, column_index, log_value):
return get_column_index(get_log_value_index(map_index, column_index), log_value) == column_index
```

Iterating through all relevant _mapping layers_ and corresponding rows is similar to how new _log values_ are added. Filtering all potential matches from all relevant rows can be done with the following funcions:
Iterating through all relevant _mapping layers_ and corresponding rows is similar to how new _log values_ are added. Filtering all potential matches from all relevant rows can be done with the following functions:

```
def get_potential_matches(log_index, map_index, log_value):
Expand Down Expand Up @@ -386,19 +386,19 @@ Storing the `log_entries` subtrees directly in their proposed merkleized format

### Alternative filter structures considered

In a search structure of a constantly growing dataset there is typically a tradeoff between the cost of adding new data and the cost of searcing the existing dataset. One extreme is just linearly storing the data, which is practically the case now with logs, with the bloom filters being mostly useless. The other extreme is one big Merkle tree with all _log values_ ever used as keys and the list of all occurences (possibly in a further merkleized format) as values. With billions of unique _log values_, adding new entries here is expected to have costs similar to that of the state, with multiple lookups and modifications/insertions at random places in a database on the order of magnitude of hundreds of gigabytes. Another issue where this is similar to the state is that removing old entries is hard and expensive. Adding logs is supposed to be cheaper than writing the state so solutions between these two extremes were considered as potentially practical, with multiple smaller structures generated periodically.
In a search structure of a constantly growing dataset there is typically a tradeoff between the cost of adding new data and the cost of searcing the existing dataset. One extreme is just linearly storing the data, which is practically the case now with logs, with the bloom filters being mostly useless. The other extreme is one big Merkle tree with all _log values_ ever used as keys and the list of all occurrences (possibly in a further merkleized format) as values. With billions of unique _log values_, adding new entries here is expected to have costs similar to that of the state, with multiple lookups and modifications/insertions at random places in a database on the order of magnitude of hundreds of gigabytes. Another issue where this is similar to the state is that removing old entries is hard and expensive. Adding logs is supposed to be cheaper than writing the state so solutions between these two extremes were considered as potentially practical, with multiple smaller structures generated periodically.

One question considered was whether to add separate keys for each unique _log value_ emitted in the given period, or to use a more compressed fixed size tree format where different _log values_ might collide (though preferably not too many of them). The second option may also include some kind of small probabilistic filter information that can help filter out the occurences of colliding _log values_ without having to access/prove the entire logs belonging to them. This decision mostly boiled down to data access efficiency, both in terms of local disk access and remote Merkle proof size. Identically structured trees can be efficiently arranged in larger units (called _epochs_ here), with values belonging to the same key in subsequent trees of an epoch located close to each other. This improves database access speed. It also allows smaller Merkle proofs with a series of leaves encoded together in an efficient format and internal nodes on only two boundary branches. Database writes are also efficient as the order of adding tree entries is not random and all the non-finalized parts of the tree can be kept in memory with a hard capped memory requirement.
One question considered was whether to add separate keys for each unique _log value_ emitted in the given period, or to use a more compressed fixed size tree format where different _log values_ might collide (though preferably not too many of them). The second option may also include some kind of small probabilistic filter information that can help filter out the occurrences of colliding _log values_ without having to access/prove the entire logs belonging to them. This decision mostly boiled down to data access efficiency, both in terms of local disk access and remote Merkle proof size. Identically structured trees can be efficiently arranged in larger units (called _epochs_ here), with values belonging to the same key in subsequent trees of an epoch located close to each other. This improves database access speed. It also allows smaller Merkle proofs with a series of leaves encoded together in an efficient format and internal nodes on only two boundary branches. Database writes are also efficient as the order of adding tree entries is not random and all the non-finalized parts of the tree can be kept in memory with a hard capped memory requirement.

The other design decision considered here was whether to hash entire logs into the list of _log value_ occurences or just store position info and have a separate tree of log entries. This does not necessarily affect local storage efficiency which should probably only store position info in the local database anyways in order to avoid duplicating log data but could still generate the hash tree based on the full log data. Though the separate _filter maps_ and log entry trees do present some additional complexity, the second option was chosen because of the size of Merkle proofs proving matches of multiple _log value_ patterns. Tests have shown that realistic log searches often yield a lot more matches for the individual _log values_ themselves that the pattern itself. Hashing entire logs into the occurence lists would mean that the proof would have to include at least the root hashes of all the individual _log value_ matches, while in the second case only the position index is needed which is more than 10x smaller with the proposed parameters.
The other design decision considered here was whether to hash entire logs into the list of _log value_ occurrences or just store position info and have a separate tree of log entries. This does not necessarily affect local storage efficiency which should probably only store position info in the local database anyways in order to avoid duplicating log data but could still generate the hash tree based on the full log data. Though the separate _filter maps_ and log entry trees do present some additional complexity, the second option was chosen because of the size of Merkle proofs proving matches of multiple _log value_ patterns. Tests have shown that realistic log searches often yield a lot more matches for the individual _log values_ themselves that the pattern itself. Hashing entire logs into the occurrence lists would mean that the proof would have to include at least the root hashes of all the individual _log value_ matches, while in the second case only the position index is needed which is more than 10x smaller with the proposed parameters.

In conclusion, for the given application the fixed tree size approach with separate position info plus probabilistic collision filter approach seemed to be the most appropriate approach. Since the _log value_ position info can be conveniently merged with the collision filter, the whole structure can be imagined as a sparse bit map on which each search operation can be thought of as applying a mask to the bit map.

### False positive rate

From the _filter maps_ a set of potential matches can be derived for any block range and _log value_ or pattern of _log values_. These matches can then be looked up in the corresponding `log_entries` trees and actually matching logs can be added to the set of results. The design guarantees that the set of potential matches includes all actual matches but and also has a consistent rate of random false positive rate.

False positives can happen when the quasi-random collision filter part of a _column index_ accidentally matches the expected value even though it was generated by a _log value_ other than the searched one. The chance of this happening is `VALUES_PER_MAP / MAP_WIDTH` per colliding enrty in a row that is relevant for the search. Assuming that most entries in a map are different from the searched one, assuming uniform random distribution of entries, the average number of colliding entries found in a relevant row is `VALUES_PER_MAP / MAP_HEIGHT`.
False positives can happen when the quasi-random collision filter part of a _column index_ accidentally matches the expected value even though it was generated by a _log value_ other than the searched one. The chance of this happening is `VALUES_PER_MAP / MAP_WIDTH` per colliding entry in a row that is relevant for the search. Assuming that most entries in a map are different from the searched one, assuming uniform random distribution of entries, the average number of colliding entries found in a relevant row is `VALUES_PER_MAP / MAP_HEIGHT`.

Though certain _log values_ might be emitted a lot more than others and therefore the _row index_ distribution might not be entirely uniform, periodical remapping of rows and using multiple _mapping layers_ ensures that over a long enough search period random collisions with more frequent _log values_ do even out. _Mapping layers_ do have another consequence though; if any row has at least `MAX_BASE_ROW_LENGTH` entries then the search logic requires looking into another row that is mapped to the searched _log value_ on the next _mapping layer_. The maximum possible number of such rows is `VALUES_PER_MAP / MAX_BASE_ROW_LENGTH` and therefore the chance of randomly hitting one is `VALUES_PER_MAP / MAX_BASE_ROW_LENGTH / MAP_HEIGHT` in the worst case. In this case an extra row has to be processed, with extra chance of finding false positives. A collision with a frequent value at a certain _mapping layer_ does not indicate a collision on the next layer though, therefore the expected number of entries in that row is no different from the first one. Having to process a third row would presume that the second one had at least `MAX_BASE_ROW_LENGTH * LAYER_COMMON_RATIO` entries. The chance of this happening after the first coincidence is practically negligible in the context of expected false positives.

Expand Down