QUESTION: How to use seastar with heterogeneous workloads?

[masscry]

Hello!

After some performance testing, I found out that shards in my application have different memory requirements, but require roughly same CPU. It looks like can't redistribute tasks evenly, it will cost a lot in terms of synchronization and passing data around. But, at same time I can't grow seastar memory indefinitely because of rather tight budget.

I have an idea to hack seastar allocator and introduce some "memory groups", to setup at engine startup - thus some dedicated shards get smaller pools, than others. Is it a valid solution? I don't want to maintain a fork, so I'll be very glad if someone can help me with ideas, how to stay on vanilla.

I've been thinking about using multiple app_templates, but as I understand there are some thread locals and other static global entities in framework, thus second app won't work as for now.

Also, with multiple apps, I will lose native ways of interaction between shards, which is also bad.

So, thank you in advance for your help.

[travisdowns]

You can also try just using the system allocator, not the seastar one. You loose some benefits (e.g., reclaim, memory statistics, "perfect" shard affinity, NUMA stuff, built-in memory profiling), but those might not matter you to you and you then get to treat all your memory as one big pool across shards.

SOTA memory allocators are in any case fast when most allocations and frees happen on the same core, so you won't lose much perf in many cases and may even gain.

Disclaimer: seastar user, not a maintainer.

[masscry]

System allocator may block. Also, in my benchmarks it is less performant.

[travisdowns]

System allocator may block. Also, in my benchmarks it is less performant.

Block off-CPU due to userspace locking, or something else? A userspace allocator with thread-local caching (or CPU-local like tcmalloc with rseq support) and a lockfree path to exchange cross-thread allocation shouldn't block in userspace.

Also, in my benchmarks it is less performant.

Interesting. Did you try only glibc default or also use a SOTA drop-in like jemalloc/tcmalloc?

I've been thinking about using multiple app_templates, but as I understand there are some thread locals and other static global entities in framework, thus second app won't work as for now.

See also #3096

[avikivity]

Multiple app_templates are one way, another is to add cpu masks to seastar::sharded constructors.

Not such asymmetric configurations require tuning and cannot adjust in runtime to changing workloads.

I guess multiple app_templates are more flexible since they run earlier. Main problem is smp::count which is static.