-
Notifications
You must be signed in to change notification settings - Fork 182
[DOC-14129]: Magma Compaction rate limiter documentation #4122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
RayOffiah
wants to merge
6
commits into
release/8.1
Choose a base branch
from
DOC-14129--8.1--Magma-Compaction-rate-limiter-documentation
base: release/8.1
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+118
−0
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
b064a1b
[DOC-14129]: Magma Compaction rate limiter documentation
RayOffiah 0e28b3f
Potential fix for pull request finding
RayOffiah 9562a87
[DOC-14129]: Magma Compaction rate limiter documentation
RayOffiah c86ac82
Potential fix for pull request finding
RayOffiah dda76ed
Potential fix for pull request finding
RayOffiah 700b8dd
[DOC-14129]: Magma Compaction rate limiter documentation
RayOffiah File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
109 changes: 109 additions & 0 deletions
109
modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| = Magma Compaction Rate Limiting | ||
| :description: This API is used to set the rate limit for Magma compaction threads. | ||
| :stem: asciimath | ||
| [abstract] | ||
| {description} | ||
|
|
||
| == Description | ||
|
|
||
| Use these API calls to set the rate limit (in bytes per second). | ||
| This setting is a global rate limit for the Memcached process shared across buckets. | ||
| The strategy for this rate limiting is to set the block waiting time on the compaction threads to maintain the specified limits. | ||
|
|
||
| Magma supports 2 types of compaction: | ||
|
|
||
| Log-Structured Merge (LSM) Tree Compactions:: | ||
| These compactions are short-running and essential for preserving the structure of the Log-Structured Merge tree. | ||
| Delaying these operations can lead to increased read latency, as the read path may need to use multiple files, resulting in higher read amplification. | ||
| If this type of compaction is lagging for level-0 of the LSM Tree, the system performs level-0 compaction within the writer threads themselves | ||
|
|
||
| Data Log Compactions:: | ||
| Data compactions can be de-prioritized compared to LSM Tree compactions. | ||
| The only consequence of slowing these operations is increased space usage. | ||
|
|
||
| == Settings | ||
|
|
||
| [cols="3,3,1"] | ||
| |=== | ||
| | Setting | Description| Range | ||
|
|
||
| | `magma_compaction_rate_limit` | ||
| | Controls the I/O bandwidth | ||
| (given in bytes per second) | ||
| that compactions across all shards consume. | ||
| Setting this value to 0 disables compaction rate limiting. | ||
| | stem:[0 … 2^64 - 1] | ||
| (default: 0) | ||
|
|
||
| | `magma_enable_compaction_dataonly_ratelimiting` | ||
| | Controls whether the compaction rate limiting runs in `data-only` mode. | ||
| (only fragment reduction compactions are rate-limited) | ||
| | `true`, `false` (default:``false``) | ||
|
|
||
| |=== | ||
|
|
||
| == HTTP Methods and URIs | ||
|
|
||
| [source, shell] | ||
| ---- | ||
| GET /pools/default/settings/memcached/global | ||
| POST /pools/default/settings/memcached/global | ||
|
|
||
| ---- | ||
|
|
||
| == Examples | ||
|
|
||
| .Retrieve the compaction limit settings | ||
| [source, shell] | ||
| ---- | ||
| curl -u Administrator:password -X GET http://localhost:8091/pools/default/settings/memcached/global | ||
| ---- | ||
|
|
||
| Running this command returns the global settings: | ||
|
|
||
| [source, json] | ||
| ---- | ||
| { | ||
| "max_connections": 20000, | ||
| "system_connections": 5000, | ||
| "magma_enable_compaction_dataonly_ratelimiting": false | ||
| } | ||
| ---- | ||
|
|
||
| .Setting the compaction rate | ||
| [source, shell] | ||
| ---- | ||
| curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \ | ||
| -d magma_compaction_rate_limit=500 | jq | ||
| ---- | ||
|
|
||
| The call executes and returns a JSON object containing the new setting. | ||
|
|
||
| [source, json] | ||
| ---- | ||
| { | ||
| "max_connections": 20000, | ||
| "system_connections": 5000, | ||
| "magma_compaction_rate_limit": 500, | ||
| "magma_enable_compaction_dataonly_ratelimiting": false | ||
| } | ||
| ---- | ||
|
|
||
| .Setting Compaction for data only rate limiting | ||
| [source,shell] | ||
| ---- | ||
| curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \ | ||
| -d magma_enable_compaction_dataonly_ratelimiting=true | jq | ||
| ---- | ||
|
|
||
| The call executes and returns a JSON object containing the new setting. | ||
|
|
||
| [source, json] | ||
| ---- | ||
| { | ||
| "max_connections": 20000, | ||
| "system_connections": 5000, | ||
| "magma_compaction_rate_limit": 500, | ||
| "magma_enable_compaction_dataonly_ratelimiting": true | ||
| } | ||
| ---- | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any guidance we can give for customers as to when they should enable rate limiting? It seems off by default, so I assume they would only enable it if they wre seeing some sort of issue?