feat: add controlled autovacuum for small databases #322
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarkResultsCurrent status |
Contributor
louiseschmidtgen
left a comment
louiseschmidtgen
left a comment
There was a problem hiding this comment.
Great work Marco! I have some little comments then we are good to go:
| import "github.com/mattn/go-sqlite3" | ||
|
|
||
| func init() { | ||
| if err := C.sqlite3_enable_autovacuum_limit(10, 16); err != C.SQLITE_OK { |
Contributor
There was a problem hiding this comment.
Are we/users going to need to tune the percentage and max pages used in auto-vacuuming?
| g.Expect(err).To(BeNil()) | ||
|
|
||
| // Expect compaction to reduce the size. | ||
| finalSize, err := server.backend.DbSize(ctx) |
Contributor
There was a problem hiding this comment.
Could we please be more precise here and check whether the size aligns with the percentage we set in qlite3_enable_autovacuum_limit?
Co-authored-by: Louise K. Schmidtgen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enables autovacuum for small databases (which includes also new ones). This allows the database to also shrink other than grow. It depends on dqlite v1.8.2.
The vacuum operation is performed after every write transaction and is controlled by two parameters (tunable through pkg/backend/sqlite/extensions.go):
max_pageswhich is the maximum number of pages to remove per write transaction. This is useful to avoid creating very big transactions. I set the default to 16 (=16 * 4KiB = 64 KiBfreed per transaction);percentwhich is the percentage (from 0 to 100) of the database to keep.The second parameter is useful to make sure that the database doesn't do unnecessary work: it is expected that the database will continuously grow and shrink, so it is best to keep some free pages for SQLite to use. I set
10%as the default. (Which means for example that is a database grows to200 MiBand then shrinks to100 MiBwe will keep the size of the storage to110 MiB).These defaults haven't been "computed" or "measured" but are what I though it would be a good default and can certainly be changed/tuned.