Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,7 @@ include::cli:partial$cbcli/nav.adoc[]
*** xref:rest-api:rest-compact-post.adoc[Performing Compaction Manually]
*** xref:rest-api:rest-autocompact-global.adoc[Auto-Compaction: Global]
*** xref:rest-api:rest-autocompact-per-bucket.adoc[Auto-Compaction: Per Bucket]
*** xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting]

** xref:rest-api:rest-rza.adoc[Server Groups API]
*** xref:rest-api:rest-servergroup-get.adoc[Getting Group Information]
Expand Down
109 changes: 109 additions & 0 deletions modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
= Magma Compaction Rate Limiting
:description: This API is used to set the rate limit for Magma compaction threads.
:stem: asciimath
[abstract]
{description}

== Description

Use these API calls to set the rate limit (in bytes per second).
This setting is a global rate limit for the Memcached process shared across buckets.
The strategy for this rate limiting is to set the block waiting time on the compaction threads to maintain the specified limits.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any guidance we can give for customers as to when they should enable rate limiting? It seems off by default, so I assume they would only enable it if they wre seeing some sort of issue?

Magma supports 2 types of compaction:

Log-Structured Merge (LSM) Tree Compactions::
These compactions are short-running and essential for preserving the structure of the Log-Structured Merge tree.
Delaying these operations can lead to increased read latency, as the read path may need to use multiple files, resulting in higher read amplification.
If this type of compaction is lagging for level-0 of the LSM Tree, the system performs level-0 compaction within the writer threads themselves

Data Log Compactions::
Data compactions can be de-prioritized compared to LSM Tree compactions.
The only consequence of slowing these operations is increased space usage.

== Settings

[cols="3,3,1"]
|===
| Setting | Description| Range

| `magma_compaction_rate_limit`
| Controls the I/O bandwidth
(given in bytes per second)
that compactions across all shards consume.
Setting this value to 0 disables compaction rate limiting.
| stem:[0 … 2^64 - 1]
(default: 0)

| `magma_enable_compaction_dataonly_ratelimiting`
| Controls whether the compaction rate limiting runs in `data-only` mode.
(only fragment reduction compactions are rate-limited)
| `true`, `false` (default:``false``)

|===

== HTTP Methods and URIs

[source, shell]
----
GET /pools/default/settings/memcached/global
POST /pools/default/settings/memcached/global

----

== Examples

.Retrieve the compaction limit settings
[source, shell]
----
curl -u Administrator:password -X GET http://localhost:8091/pools/default/settings/memcached/global
----

Running this command returns the global settings:

[source, json]
----
{
"max_connections": 20000,
"system_connections": 5000,
"magma_enable_compaction_dataonly_ratelimiting": false
}
----

.Setting the compaction rate
[source, shell]
----
curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \
-d magma_compaction_rate_limit=500 | jq
----

The call executes and returns a JSON object containing the new setting.

[source, json]
----
{
"max_connections": 20000,
"system_connections": 5000,
"magma_compaction_rate_limit": 500,
"magma_enable_compaction_dataonly_ratelimiting": false
}
----

.Setting Compaction for data only rate limiting
[source,shell]
----
curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \
-d magma_enable_compaction_dataonly_ratelimiting=true | jq
----

The call executes and returns a JSON object containing the new setting.

[source, json]
----
{
"max_connections": 20000,
"system_connections": 5000,
"magma_compaction_rate_limit": 500,
"magma_enable_compaction_dataonly_ratelimiting": true
}
----
8 changes: 8 additions & 0 deletions modules/rest-api/partials/rest-memory-and-storage-table.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,12 @@
| `POST`
| `/pools/default/buckets/[bucket-name]`
| xref:rest-api:rest-autocompact-per-bucket.adoc[Auto-Compaction: Per Bucket]

| `GET`
| `/pools/default/settings/memcached/global`
| xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting]

| `POST`
| `/pools/default/settings/memcached/global`
| xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting]
|===