[Bug] soci-snapshotter-grpc.service takes up huge memory

[prafgup]

Description

I was trying to profile the memory usage by SOCI locally as well as on a k8s node and figured that the memory usage by soci keeps on increasing even with no pods running on the node.

I have tried using containerd v1.7.27 and v2.2.0 which both having the same issue.

Command to check the usage by system.slice/soci-snapshotter-grpc.service

systemd-cgtop -m

Output -

^ see above that soci is using 2.6+ GB without any containers running currently using soci.

ls -lah /var/lib/soci-snapshotter-grpc

total 72M
drwxr-xr-x  6 root root 4.0K Jan 19 18:38 .
drwxr-xr-x 56 root root 4.0K Jan 19 18:41 ..
drwxr-xr-x  4 root root 4.0K Jan 20 17:58 content
-rw-------  1 root root  88M Jan 20 18:26 metadata.db
drwx------  3 root root 4.0K Jan 19 18:38 snapshotter
drwx------  4 root root 4.0K Jan 20 17:58 soci
drwx------  2 root root 4.0K Jan 19 18:38 unpack



sudo du -h --max-depth=2

4.0K	./unpack
11G	./snapshotter/snapshots
11G	./snapshotter
2.0G	./soci/spancache
180K	./soci/httpcache
2.0G	./soci
263M	./content/blobs
4.0K	./content/ingest
263M	./content
13G	.

I am using a basic soci config as such -

 cat /etc/soci-snapshotter-grpc/config.toml


[http]
  DialTimeoutMsec = 10000
  ResponseHeaderTimeoutMsec = 10000
  RequestTimeoutMsec = 300000
  MaxRetries = 10
  MinWaitMsec = 300
  MaxWaitMsec = 300000


[cri_keychain]
enable_keychain = true
image_service_path = "/run/containerd/containerd.sock"

This would to be an issue where a system would be running multiple unique images. The ram usage keeps on increasing as well as cpu usage (even after pods were deleted).

sudo systemctl status soci-snapshotter-grpc
● soci-snapshotter-grpc.service - SOCI Snapshotter gRPC
     Loaded: loaded (/etc/systemd/system/soci-snapshotter-grpc.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-01-19 18:38:18 UTC; 24h ago
   Main PID: 924 (soci-snapshotte)
      Tasks: 107 (limit: 75875)
     Memory: 2.6G
        CPU: 6min 58.194s
     CGroup: /system.slice/soci-snapshotter-grpc.service
             └─924 /usr/local/bin/soci-snapshotter-grpc --address /run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock --config /etc/soci-snapshotter-grpc/config.toml --log-level info

Steps to reproduce the bug

1. Launch soci-snapshotter-grpc as a systemd service
2. Check memory usage systemd-cgtop -m (soci would be using ~10mb)
3. Pull a soci image and run it (use a pod for k8s). (memory spikes to 100-200mb)
4. Wait for sometime while the background spans are fetched (memory keeps on spiking to 1-2GB, the whole image is less than the memory usage)
5. [Repeat step 3 multiple times if one wants]
6. Delete the pod (memory is still not released)

Describe the results you expected

I expect soci to not have constant high memory usage and memory to be released once the pod is deleted. Soci should cache the blobs on disk and not keep it in memory, as the memory would keep on filling up with new unique chunks and soon OOM.

Host information

1. OS: x86_64
2. Snapshotter Version: v0.12.0
3. Containerd Version: v1.7.27 and v2.2.0

Any additional context or information about the bug

No response

[Kern--]

The memory usage reported by the cgroup includes more than the memory that the program has allocated, most notably the page cache.

On my instance (after running/exiting a 60MB compressed nginx container) I see ~120MB of memory associated with the cgroup:

$ sudo systemctl status soci-snapshotter.service 
● soci-snapshotter.service - soci snapshotter containerd plugin
     Loaded: loaded (/usr/lib/systemd/system/soci-snapshotter.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-01-20 23:44:36 UTC; 6min ago
TriggeredBy: ○ soci-snapshotter.socket
       Docs: https://github.com/awslabs/soci-snapshotter
   Main PID: 580631 (soci-snapshotte)
      Tasks: 30 (limit: 37837)
     Memory: 121.5M
        CPU: 1.819s
     CGroup: /system.slice/soci-snapshotter.service
             └─580631 /usr/bin/soci-snapshotter-grpc

But only ~20MB of that is used by the snapshotter while the other ~100 MB is in the page cache.

$ cat /sys/fs/cgroup//system.slice/soci-snapshotter.service/memory.stat
anon 21053440
file 101703680
kernel 1900544
... Truncated for brevity

The page cache reclaimable if your system has memory pressure and it does automatically go away once the image is deleted and the snapshots are unmounted by containerd's GC:

$ sudo systemctl status soci-snapshotter.service 
● soci-snapshotter.service - soci snapshotter containerd plugin
     Loaded: loaded (/usr/lib/systemd/system/soci-snapshotter.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-01-20 23:44:36 UTC; 12min ago
TriggeredBy: ○ soci-snapshotter.socket
       Docs: https://github.com/awslabs/soci-snapshotter
   Main PID: 580631 (soci-snapshotte)
      Tasks: 30 (limit: 37837)
     Memory: 32.9M
        CPU: 1.921s
     CGroup: /system.slice/soci-snapshotter.service
             └─580631 /usr/bin/soci-snapshotter-grpc

Are you seeing any impact from this memory usage or is it just concerning that it's so high?

[Kern--]

For reference, this container still works if I set the SOCI Snapshotter's max memory to 50M, but it's slower since more reads/writes go to disk:

$ sudo systemctl status soci-snapshotter.service 
● soci-snapshotter.service - soci snapshotter containerd plugin
     Loaded: loaded (/usr/lib/systemd/system/soci-snapshotter.service; enabled; preset: disabled)
     Active: active (running) since Wed 2026-01-21 00:31:39 UTC; 10s ago
TriggeredBy: ○ soci-snapshotter.socket
       Docs: https://github.com/awslabs/soci-snapshotter
   Main PID: 587119 (soci-snapshotte)
      Tasks: 26 (limit: 37837)
     Memory: 49.7M (max: 50.0M available: 232.0K)

[prafgup]

Thanks for the explanation, I do see SOCI was using ~251 MB service memory and ~3.5 GB file cache.

sudo cat /sys/fs/cgroup/system.slice/soci-snapshotter-grpc.service/memory.stat
anon 263188480
file 3767799808

After force cleaning the page cache -

sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

sudo cat /sys/fs/cgroup/system.slice/soci-snapshotter-grpc.service/memory.stat
anon 154050560
file 120639488

It now uses ~147.0 MB service memory and ~115.0 MB of file cache.

Are you seeing any impact from this memory usage or is it just concerning that it's so high?

I don't see any impact yet but just concerned that it would keep on increasing in the k8s node with more and more pods we launch with unique images. Thus limiting the memory for other services.
I believe these unused images would be cleaned up by the kubelet in case we would be running low on memory - https://kubernetes.io/docs/concepts/architecture/garbage-collection/#containers-images
Was wondering if memory usage keeps on increasing, would there be a node wide page cache drop or kublet cleaning the images 🤔

Do we have profiling/metrics over how much slower would soci be if page cache gets cleaned up due to system constraints and read/write would start going through disk?

Just a added note, I profiled using stargz as well and seems like it behaves similar to soci (similar memory profile and similar cleanup behavior) -

sudo cat /sys/fs/cgroup/system.slice/stargz-snapshotter.service/memory.stat
anon 215154688
file 2454671360
kernel 159047680

[Shubhranshu153]

Do we have profiling/metrics over how much slower would soci be if page cache gets cleaned up due to system constraints and read/write would start going through disk? -> This probably is a hard question to answer deterministically as it will be dependent on type of file operation that needs to be done that is slowed down by the fuse fs. Thus it will be application dependent and probably a good test for your application will be by limiting the memory allowed to soci and doing a performance benchmark.

With respect to page cache it is handled by the kernel as a shared resource and reclaimed whenever memory thresholds are reached. The exact numbers i am not sure how its configured.

Does it answer your question?

[prafgup]

Does it answer your question?

Not entirely, I'm wondering if there are any published benchmarks comparing SOCI to other snapshotters (particularly stargz) beyond just pull/index times. Specifically, metrics like:

• CPU and memory usage
• Maximum concurrent mounts
• Performance under memory pressure
• I/O overhead from FUSE
• Possible limitation over number of FUSE mounts (with growing number of SOCI pulled images in a system)
• etc..

I think these would be valuable for early adopters evaluating snapshotters for production use. If no such report exists, it might be worth noting as a documentation gap. Imo it does not makes much sense for each organization to benchmark these generic metrics before adopting SOCI which should have been already made available (maybe by AWS Fargate?)

[Shubhranshu153]

Thank you for the feedback. The benchmarking metrics in the desired granularity are not published publicly as such we can take it as a documentation gap. Usually what i have observed early adopters for soci monitor startup and application performance for their top x% of workloads in the dimension of startup time and application performance metrics. Having said that it can be expensive for organizations and having a baseline metrics in certain standardized dimensions can help for calibration.

Ticket: #1855