add byte limit implementation for cache by p0mvn · Pull Request #40 · osmosis-labs/iavl

p0mvn · 2022-04-02T02:27:36Z

Background

This PR introduces byte limit implementation for cache and switches the fast cache to this implementation. It introduces a small decrease in performance in exchange for modular and reusable design as well as the ability to keep track of the number of bytes used.

Deployed nodes with 50MB, 75MB, 100MB, and 150MB fast cache. The nodes are stable after running for a day, and I'm waiting to have more time to understand the RAM usage with this change better.

Implementation Details

Refactored cache module to have a general LRU cache implementation. Introduced node and byte limit decorators to wrap around LRU cache and provide a composable design.

osmosis-labs/osmosis#1187

Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-4    4.32µs ± 6%    4.29µs ± 5%    ~     (p=0.768 n=5+10)
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-4    16.1µs ± 3%    16.9µs ± 1%  +4.87%  (p=0.002 n=5+8)
Medium/goleveldb-100000-100-16-40/query-hits-fast-4                     611ns ± 9%     606ns ±12%    ~     (p=0.679 n=5+10)
Medium/goleveldb-100000-100-16-40/query-hits-slow-4                    21.9µs ± 5%    22.6µs ± 6%    ~     (p=0.075 n=5+10)
Medium/goleveldb-100000-100-16-40/iteration-fast-4                     82.1ms ± 6%    78.4ms ± 8%    ~     (p=0.099 n=5+10)
Medium/goleveldb-100000-100-16-40/iteration-slow-4                      1.92s ± 1%     1.96s ± 3%  +2.18%  (p=0.020 n=4+9)
Medium/goleveldb-100000-100-16-40/update-4                              247µs ± 8%     251µs ±12%    ~     (p=0.768 n=5+10)
Medium/goleveldb-100000-100-16-40/block-4                              31.3ms ± 8%    31.6ms ± 4%    ~     (p=0.513 n=5+10)

name                                                                 old alloc/op   new alloc/op   delta
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-4      814B ± 0%      814B ± 0%    ~     (all equal)
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-4    1.40kB ± 1%    1.41kB ± 1%  +0.70%  (p=0.018 n=5+10)
Medium/goleveldb-100000-100-16-40/query-hits-fast-4                     0.00B          0.00B         ~     (all equal)
Medium/goleveldb-100000-100-16-40/query-hits-slow-4                    2.00kB ± 0%    1.99kB ± 1%    ~     (p=0.117 n=5+10)
Medium/goleveldb-100000-100-16-40/iteration-fast-4                     29.3MB ± 0%    29.3MB ± 0%    ~     (p=0.513 n=5+10)
Medium/goleveldb-100000-100-16-40/iteration-slow-4                      276MB ± 0%     276MB ± 0%    ~     (p=0.210 n=4+8)
Medium/goleveldb-100000-100-16-40/update-4                             46.5kB ± 3%    47.5kB ± 4%    ~     (p=0.129 n=5+10)
Medium/goleveldb-100000-100-16-40/block-4                              5.43MB ± 1%    5.65MB ± 8%    ~     (p=0.129 n=5+10)

name                                                                 old allocs/op  new allocs/op  delta
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-4      16.0 ± 0%      16.0 ± 0%    ~     (all equal)
Medium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-4      24.0 ± 0%      24.4 ± 2%    ~     (p=0.308 n=5+10)
Medium/goleveldb-100000-100-16-40/query-hits-fast-4                      0.00           0.00         ~     (all equal)
Medium/goleveldb-100000-100-16-40/query-hits-slow-4                      34.0 ± 0%      34.0 ± 0%    ~     (all equal)
Medium/goleveldb-100000-100-16-40/iteration-fast-4                       523k ± 0%      523k ± 0%    ~     (p=0.354 n=5+10)
Medium/goleveldb-100000-100-16-40/iteration-slow-4                      4.71M ± 0%     4.71M ± 0%  +0.01%  (p=0.004 n=4+8)
Medium/goleveldb-100000-100-16-40/update-4                                486 ± 7%       516 ±11%    ~     (p=0.053 n=5+10)
Medium/goleveldb-100000-100-16-40/block-4                               60.4k ± 1%     62.0k ± 4%    ~     (p=0.099 n=5+10)

name                                      old time/op    new time/op    delta
Add/small_-_limit:_10K,_key_size_-_10b-4    1.09µs ± 6%    1.08µs ± 3%    ~     (p=0.937 n=5+5)
Add/med_-_limit:_100K,_key_size_20b-4       1.29µs ± 1%    1.31µs ± 0%  +1.37%  (p=0.024 n=5+5)
Add/large_-_limit:_1M,_key_size_30b-4       1.38µs ±10%    1.39µs ±11%    ~     (p=0.841 n=5+5)

name                                      old alloc/op   new alloc/op   delta
Add/small_-_limit:_10K,_key_size_-_10b-4      106B ± 1%      106B ± 0%    ~     (p=0.333 n=5+4)
Add/med_-_limit:_100K,_key_size_20b-4         127B ± 0%      127B ± 0%    ~     (all equal)
Add/large_-_limit:_1M,_key_size_30b-4         138B ± 1%      138B ± 1%    ~     (p=1.000 n=4+4)

name                                      old allocs/op  new allocs/op  delta
Add/small_-_limit:_10K,_key_size_-_10b-4      4.00 ± 0%      4.00 ± 0%    ~     (all equal)
Add/med_-_limit:_100K,_key_size_20b-4         4.00 ± 0%      4.00 ± 0%    ~     (all equal)
Add/large_-_limit:_1M,_key_size_30b-4         4.00 ± 0%      4.00 ± 0%    ~     (all equal)

fast_node_test.go

alexanderbez · 2022-04-04T17:54:37Z

fast_node.go

 	return fn.key
 }

+func (fn *FastNode) GetFullSize() int {


Can we add a godoc describing the value returned here? It's not super clear to me.

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

ValarDragon · 2022-04-07T01:30:07Z

cache/bytes_limit_decorator.go

+type lruCacheWithBytesLimit struct {
+	lruCache
+	bytesLimit       int
+	curBytesEstimate int


why is it called estimate? Should we add a comment that it may potentially slightly undercount, but we deem it fine?

It is called estimate because I did not find a straightforward way to test actual allocations.

I tried grabbing memory from runtime to test but it was slightly off. I think it's because it doesn't produce the memory needed for allocating "slice metadata" - length, capacity and pointer to data.

This estimate is based on the knowledge of how slices and strings are represented in memory in Go. That's why I named it estimate.

I'll add a comment about this

Oh gotcha. Yeah being off by a small constant for go memory layout details (which aren't guaranteed to be preserved across versions) is totally fine!

The comment is added

ValarDragon · 2022-04-07T01:32:10Z

cache/cache.go

+const (
+	LRU             Type = 0
+	LRU_node_limit  Type = 1
+	LRU_bytes_limit Type = 2
+)


thoughts on switching this to iota syntax? https://yourbasic.org/golang/iota/

Also are we using all three types?

We are using the last 2 in iavl, and the first one is the abstract implementation of the last 2.

I made it so that it can't be initialized outside of cache package. However, it still needs to have its own GetType() method

Do we need the first one to be implemented? Perhaps we make a follow-up issue to delete it?

Ohh the point is its needed, since we're doing things decorator style.

I guess I don't understand the type enum + decorator syntax combination. I don't think it needs to block the PR, but maybe we should make a follow-up issue about it

The decorator pattern implies that LRU cache (the underlying implementation) can still be used on its own. We just don't use it in IAVL. Decorators are wrappers around the main abstraction to provide an additional layer of functionality. In our case, this functionality is limiting the cache.

We have three types implemented:

type Type int const ( LRU Type = 0 LRU_node_limit Type = 1 LRU_bytes_limit Type = 2 )

Only the last 2 are used in IAVL directly. However, this is composable by design - if we want the limit to be removed in the future - we just swap the cache for the regular LRU type

Let me know if this makes sense

ValarDragon · 2022-04-15T03:37:36Z

nodedb.go

 	defaultStorageVersionValue = "1.0.0"
 	fastStorageVersionValue    = "1.1.0"
-	fastNodeCacheLimit = 100000
+	fastNodeCacheLimit         = 100 * 1024 * 1024


we should rename this to fastNodeCacheBytesLimit right?

faddat · 2022-04-18T14:47:42Z

I am working on speedruns for db comparisons. So I am going to be a teensy bit bold and backport this now. I will report back on if it successfully resolves the issue that I am hitting with rocksdb (basically v6 becomes extremely ram hungry and I've not been able to do anything to change that.)

Previous state: Nodes would reach 128gb and be killed by the oom reaper

New sate: unknown

Speaking unscientifically, the update seems much faster.

p0mvn added 4 commits April 1, 2022 22:41

move lruCacheNodeLimit implementation into a separate file

1b00d89

rename cache_test to cache_node_limit_test.go

86f8f5e

implment and test GetFullSize for fast nodes

a9e8eaa

refactor cache to minimize duplication between various implementations

2037ed2

p0mvn mentioned this pull request Apr 2, 2022

Implement configurable fast node cache size osmosis-labs/osmosis#1163

Closed

8 tasks

p0mvn added 16 commits April 2, 2022 23:51

implement bytes limit cache, add simple unit tests for add

c9b0397

rename decorators

bd8273b

add function to test current bytes

557613c

finish bytes limit add tests and fmt

16cc236

do not return updated elements

7645c4c

add godoc

652d9f0

switch fast node cache to 100MB

31097db

add cache tests to makefile and CI

c3787d0

fix 32 bit test in fast_node_test.go

d53703f

restore Makefile and ci.yml - should still run cache tests

80febdc

unit test bytes limit remove

734412b

reuse code between tests

0b39cff

fmt

0363e41

refactor add tests to reuse code

9b3ccb4

add benchmarks

edec9c3

fmt

6e5b9cd

p0mvn marked this pull request as ready for review April 4, 2022 05:07

p0mvn requested review from ValarDragon and alexanderbez April 4, 2022 16:17

alexanderbez reviewed Apr 4, 2022

View reviewed changes

fast_node_test.go Outdated Show resolved Hide resolved

alexanderbez reviewed Apr 4, 2022

View reviewed changes

p0mvn and others added 2 commits April 4, 2022 11:52

Update fast_node_test.go

fec7c9a

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

add godocs for cache.Node implementations and fmt

b9ee524

ValarDragon reviewed Apr 7, 2022

View reviewed changes

This was referenced Apr 12, 2022

Potential RAM Leak osmosis-labs/osmosis#1140

Closed

port bytes size iavl cache to v6.x to mitigate RAM issues osmosis-labs/osmosis#1239

Closed

ValarDragon reviewed Apr 15, 2022

View reviewed changes

rename fastNodeCacheLimit to fastNodeCacheLimitBytes

076dc56

faddat mentioned this pull request Apr 18, 2022

6.x maitnenance osmosis-labs/osmosis#1281

Closed

4 tasks

add comment about curBytesEstimate

b59ba72

p0mvn mentioned this pull request Jun 16, 2022

refactor: implement cache abstraction and unit test cosmos/iavl#506

Merged

Conversation

p0mvn commented Apr 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

faddat commented Apr 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

p0mvn commented Apr 2, 2022 •

edited

Loading

faddat commented Apr 18, 2022 •

edited

Loading