Garnet crashes on large dataset ingestion (20K records) with high CPU spike

[Saqibs575]

Describe the bug

GarnetServer Exception with Large Dataset (20K SQL Records) + High CPU Usage

I am running GarnetServer.exe (GarnetServer 1.1.6+c81684f5b216fb7e70e5f16e806842b54ae57ee9
) on Windows, and I am encountering a critical exception when processing large datasets (~20,000 SQL records).

The issue does not occur with small datasets, but consistently appears when the payload size increases.

Additionally, CPU usage spikes significantly (up to ~87–89%) during retries and processing.

---

Error Logs

When processing large data, the server throws the following exception:

12::12::10 crit: Session[0] [127.0.0.1:58660] [02F36FD6] ProcessMessages threw an exception: Tsavorite.core.TsavoriteException: Entry does not fit on page at Tsavorite.core.TsavoriteLog.Enqueue[THeader,TInput,TEpochAccessor](THeader userHeader, SpanByte& item1, TInput& input, TEpochAccessor epochAccessor, Int64& logicalAddress) in //libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLog.cs:line 1043 at Garnet.server.ObjectSessionFunctions.WriteLogRMW[TEpochAccessor](Byte[]& key, ObjectInput& input, Int64 version, Int32 sessionID, TEpochAccessor epochAccessor) in //libs/server/Storage/Functions/ObjectStore/PrivateMethods.cs:line 61 at Garnet.server.ObjectSessionFunctions.PostRMWOperation[TEpochAccessor](Byte[]& key, ObjectInput& input, RMWInfo& rmwInfo, TEpochAccessor epochAccessor) in //libs/server/Storage/Functions/ObjectStore/RMWMethods.cs:line 293 at Garnet.server.StorageSession.RMWObjectStoreOperation[TObjectContext](Byte[] key, ObjectInput& input, ObjectOutputHeader& output, TObjectContext& objectStoreContext) in //libs/server/Storage/Session/ObjectStore/Common.cs:line 31 at Garnet.server.StorageSession.ListPush[TObjectContext](Byte[] key, ObjectInput& input, ObjectOutputHeader& output, TObjectContext& objectStoreContext) in //libs/server/Storage/Session/ObjectStore/ListOps.cs:line 363 at Garnet.server.GarnetApi`3.ListLeftPush(Byte[] key, ObjectInput& input, ObjectOutputHeader& output) in //libs/server/API/GarnetApiObjectCommands.cs:line 258 at Garnet.server.RespServerSession.ListPush[TGarnetApi](RespCommand command, TGarnetApi& storageApi) in //libs/server/Resp/Objects/ListCommands.cs:line 45 at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](RespCommand cmd, TGarnetApi& storageApi) in //libs/server/Resp/RespServerSession.cs:line 922 at Garnet.server.RespServerSession.ProcessMessages() in //libs/server/Resp/RespServerSession.cs:line 639 at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in //libs/server/Resp/RespServerSession.cs:line 484

Environment Details

Garnet

• Version: 1.1.6
• OS: Windows (64-bit)
• Command:

GarnetServer.exe --config-import-path garnet.conf

Redis Client

• Library: redis-py
• Version: 6.2.0
• Connection: localhost:6379

Machine Details

• OS: Windows 10
• RAM: 32GB

---

Observations

• Works fine for small datasets
• Issue occurs when dataset size increases (~20K records)
• Likely related to:
  • Tsavorite page size limitation
  • Large payload per RMW/List operation
  • Log segment overflow or serialization size constraints
  • CPU spikes heavily during retry logic loops

---

Additional Notes

• I have implemented retry logic, but it increases CPU usage significantly during failures
• Task Manager screenshots confirm Garnet CPU usage reaching ~87–89%

---

Request

Please advise on:

• Maximum safe payload size per operation
• Recommended batching strategy for large datasets
• Any configuration tuning to avoid:
  • Entry does not fit on page exception
  • High CPU usage under bulk load

Steps to reproduce the bug

1. Start Garnet server:

GarnetServer.exe --config-import-path garnet.conf

2. Connect using redis client (redis-py)
3. Insert ~20,000 SQL-derived records into a list using LPUSH / list operations
4. Observe failure during bulk ingestion

Expected behavior

Garnet should handle large dataset inserts (20K SQL records) without:

• Crashing sessions
• Throwing TsavoriteException: Entry does not fit on page
• Excessive CPU usage spikes

Screenshots

Task Manager CPU usage screenshots (Garnet process ~87–89%)

Release version

No response

IDE

No response

OS version

No response

Additional context

Configuration Details
{
//
/ Garnet Configuration File /
//
/* Use this file to configure any values different from the defaults specified in libs/host/defaults.conf /
/ /
/ Example: /
/ "Port" : 1234, /
/ "Address" : "1.2.3.4", /
/ "MemorySize" : "32g", /
/ ... /
/ /
/ In order to use this file, run GarnetServer with the --config-import-path command line argument /
/ (e.g. GarnetServer.exe --config-import-path /path/to/garnet.conf). /
/*********************************************************************************************************/
"AuthenticationMode": "Password",
"Password": password",

"MemorySize": "16g",

"CheckpointDir": "checkPoinDir",

"EnableAOF": true,
"Recover": true,
"CommitFrequencyMs": 1000
}

[badrishc]

The AofPageSize should be configured to be at least twice PageSize. If you are using default PageSize of 32m, then configure AofPageSize to 64m.

[Saqibs575]

Questions Regarding Large Payload Handling & Checkpoint Growth

First of all, thank you @badrishc for the help regarding the previous issue.

The following configuration is now working correctly for my large payload workload after your guidance:

{
  "PageSize": "32m",
  "AofPageSize": "64m",
  "AofMemorySize": "128m"
}

Initially, I only changed AofPageSize as suggested:

"The AofPageSize should be configured to be at least twice PageSize."

However, Garnet still failed to start with the following error:

Unable to initialize server due to exception:
Memory size (67108864) must be at least twice the page size (67108864)

The issue was resolved only after I additionally configured:

"AofMemorySize": "128m"

After adding this, large payload operations started working properly.

I had actually encountered this issue earlier as well, but at the time I could not identify the root cause because the error message was somewhat unclear. The message references only "page size" and "memory size", while the actual constraint in this case was between AofPageSize and AofMemorySize.

So thank you again for the clarification and support.

---

Additional Context

This issue may be related to the earlier discussion here: #630

As suggested there, I already switched to:

"CompactionType": "Lookup"

I have also already tested various memory/compaction-related configurations (page sizes, AOF tuning, overwrite/delete patterns, storage-tier settings, etc.), but memory usage still continues increasing over time.

---

Current Concern

I am coming from Redis, and one thing I find difficult in Garnet is the amount of manual tuning required for large payloads.

In Redis, I typically do not need to configure:

• page sizes
• AOF page sizes
• AOF memory limits

based on payload size.

Redis automatically handles large values internally without requiring users to estimate upper bounds in advance.

With Garnet, however, if these settings are not properly tuned:

• operations fail with:

  Entry does not fit on page
  

• or large payloads become unstable.

---

Request

Would it be possible to provide Automatic Sizing Mode?

where Garnet automatically grows internal page sizes when encountering larger payloads.

It can internally derives:

• PageSize
• AofPageSize
• AofMemorySize

instead of requiring users to manually tune all related parameters.

---

Another Concern — Checkpoint / AOF Disk Growth

I also observed another issue during large blob overwrite/delete workloads.

Example workload

Python test:

def mb_payload(mb: int) -> bytes:
    return b"x" * (mb * 1024 * 1024)

for i in range(100):
    redis_client.set("test_key", mb_payload(100))

Since the same key is overwritten repeatedly, I expected disk usage to stay relatively close to the latest live dataset size.

However:

• checkpointDir grows significantly
• deleted/overwritten records continue occupying storage
• even after deleting keys, storage usage remains high

I understand temporary amplification during compaction/checkpointing is expected, but I am observing persistent storage growth even after:

• deletes
• restart/recovery

---

Current Configuration

{
  "AuthenticationMode": "Password",
  "Port": 6380,

  "CheckpointDir": "checkpoint",

  "EnableAOF": true,
  "Recover": true,

  "PageSize": "32m",
  "AofPageSize": "64m",
  "AofMemorySize": "128m",

  "MemorySize": "32g",

  "AofSizeLimit": "512m",

  "CompactionType": "Lookup",
  "CompactionForceDelete": true,

  "CommitFrequencyMs": 1000
}

---

Questions

1. Is this storage growth expected for repeated overwrite/delete workloads with huge payloads?

2. Is there a recommended way to reclaim:

   • stale checkpoint files
   • deleted blob storage
   • old AOF segments

3. Does Garnet currently support behavior similar to Redis AOF rewrite where storage eventually converges close to the live dataset size?

4. Are there additional settings recommended specifically for:

   • very large payloads (100MB+)
   • repeated overwrite/delete scenarios
   • persistent storage optimization

---

Additional Feedback

Overall, Garnet performance is good.

The main challenge currently is discoverability and tuning complexity around:

• PageSize
• AofPageSize
• AofMemorySize
• compaction behavior
• checkpoint cleanup

especially for users migrating from Redis where these details are mostly abstracted away.

Thank you again for the support and guidance.