smp: add a function that barriers memory prefault work

[tomershafir]

Currently, memory prefault logic is internal and seastar doesnt provide much control to users. In order to improve the situation, I suggest to provide a barrier for the prefault threads. This allows to:

• Prefer predictable low latency and high throughput from the start of request serving, at the cost of a startup delay, depending on machine characteristics and application specific requirements. For example, a fixed capacity on prem db setup, where slower startup can be tolerated. From users perspective, they generally cannot tolerate inconsistency (like spikes in latency).
• Similarly, improve user scheduling decisions, like running less critical tasks while prefault works.
• Reliably test the prefault logic, improving reliability and users trust in seastar.

I tested locally. If you approve, next I will try to submit a prefault test.

[avikivity]

Did you observe latency impact from the prefault threads? It was written carefully not to have latency impact, but it's of course possible that some workloads suffer.

[tomershafir]

As you described in #1702, page faults can cause deviation, and following up the example, there can be 25sec where latency is variably higher.

[avikivity]

As you described in #1702, page faults can cause deviation, and following up the example, there can be 25sec where latency is variably higher.

I said nothing about latency being higher there.

We typically run large machines with a few vcpus not assigned to any shards, and the prefault threads run with low priority.

[tomershafir]

There are 2 aspects:

1. Page faults

In the previous comment, I meant page fault latency. The page faults can cause high latency unpredictably until the prefaulter finishes.

Regarding page faults measurement, it seems I cannot reliably measure on my env.

2. Prefault threads competition

I tried to non scientifically isolate wall time overhead of prefault threads:

I have a test app that performs file I/O and process memory buffers repeatedly. I used Ubuntu Orbstack VM with 1 NUMA node, 10 cores, --memory=14G - effectively a small NUMA node, and a small input to let the overhead be most visible.

• With --lock-memory=1 without waiting, I see that the chrono time of the actual work is significantly higher than with --lock-memory=0. (~1800ms > ~600ms)
• When waiting before doing actual work, I see that the overhead is removed.
• When building seastar without prefault code and --lock-memory=1 I dont see the overhead.

[tomershafir]

By default seastar uses all vcpus, which makes sense for resource efficiency.

Also, do you free specific vcpus? Like one per numa node, the granularity of prefault threads.

[avikivity]

By default seastar uses all vcpus, which makes sense for resource efficiency.

Also, do you free specific vcpus? Like one per numa node, the granularity of prefault threads.

1 in 8, with NUMA awareness. They're allocated for kernel network processing. See perftune.py.

[tomershafir]

Nice. Let me know if this change makes sense to you

[tomershafir]

@avikivity ping

[tomershafir]

I also tried to simulate perftune with 1 free vcpu: --cpuset=0-8 given the above setup, and I still observe the overhead, even though its less (~1600ms).

[avikivity]

I don't understand what this 1600ms overhead is.

[tomershafir]

I tried to non scientifically isolate wall time overhead of prefault threads:

I have a test app that performs file I/O and process memory buffers repeatedly. I used Ubuntu Orbstack VM with 1 NUMA node, 10 cores, --memory=14G - effectively a small NUMA node, and a small input to let the overhead be most visible.

• With --lock-memory=1 without waiting, I see that the chrono time of the actual work is significantly higher than with --lock-memory=0. (~1800ms > ~600ms)
• When waiting before doing actual work, I see that the overhead is removed.
• When building seastar without prefault code and --lock-memory=1 I dont see the overhead.

I mean its the wall time of the work that I observe, with 1 free vcpu: --cpuset=0-8 given the above setup.

[avikivity]

I tried to non scientifically isolate wall time overhead of prefault threads:
I have a test app that performs file I/O and process memory buffers repeatedly. I used Ubuntu Orbstack VM with 1 NUMA node, 10 cores, --memory=14G - effectively a small NUMA node, and a small input to let the overhead be most visible.

• With --lock-memory=1 without waiting, I see that the chrono time of the actual work is significantly higher than with --lock-memory=0. (~1800ms > ~600ms)
• When waiting before doing actual work, I see that the overhead is removed.
• When building seastar without prefault code and --lock-memory=1 I dont see the overhead.

I mean its the wall time of the work that I observe, with 1 free vcpu: --cpuset=0-8 given the above setup.

Okay. But what's the problem with that time?

Anyway, is we add future<> seastar::wait_for_background_initialization() we can have application startup code elect to wait for it before opening ports. This way it can let its own initialization work overlap with memory initialization.

[tomershafir]

The problem is that it is slower and not consistent/predictable. After memory initialization it is faster and consistent.

Regarding the implementation, the problem is that pthread_join blocks, so in the current implementation can't using a future be misleading?

[avikivity]

How is pthread_join relevant?

[tomershafir]

Currently, the logical barrier waits for pthread_join on all the threads that perform the prefault work. It will block the reactor thread.

[avikivity]

Ah, you're referring to the patch while I was referring to the current state. Don't use join then, instead figure out something else that can satisfy a seastar::promise. Maybe it's as simple as seastar::alien::submit_to(0, [&] { _reactor._prefault_complete.set_value(); }).

[tomershafir]

ah, I see. So if I understand correctly, you have just restated the clarified motivation for the patch (pls correct me if I'm wrong). I'll work on a non-blocking method next week.

[avikivity]

I don't completely see that it's useful but can't deny that it might be.

I'd be happier with an example of a real application requiring it.

[tomershafir]

I have only a test application. How about scylladb?

[avikivity]

I have only a test application. How about scylladb?

I'm not aware of reports of problem during the prefault stage. It takes some time for a node to join the cluster, and by that time enough memory was prefaulted for it to work well.

[tomershafir]

sorry for the delay and the confusion. I saw you added a join method: #2679 Ill try to rebase my changes

[tomershafir]

v2:

Rebase master and rewrite a non-blocking seastar::join_memory_prefault

[tomershafir]

@avikivity hey, pls review

[avikivity]

This warning isn't helpful to users; what can they do?

[tomershafir]

They should fix the configuration to make Seastar actually prefault memory, or remove the redundant join if not intended. But if you prefer otherwise, Ill remove it.

[avikivity]

The application doesn't know if the user wants to prefault memory or not. It's common in local testing not to prefault, and in production to prefault.

[avikivity]

Instead of this, you can promise::set_value() on the promise if you don't initialize the prefaulter.

[tomershafir]

I agree, this way the promise is not left unresolved. Will call it on smp::configure.

[avikivity]

Alternatively, we can document that that the join() should only be run on shard 0. I expect that most applications run initialization code in shard 0 and don't need it to be available anywhere else.

(I want to deprecate and remove run_in_background, I think it's dangerous)

[tomershafir]

Why not let it work on any shard? We anyway need the new promise member on the reactor class.

Also, maybe make run_in_background internal instead? its needed for such use cases.

[tomershafir]

wdyt?

[tomershafir]

@avikivity ping

[avikivity]

It seems an unnecessary complication. Applications typically have a main thread that runs on shard 0 that coordinates the startup process.

[tomershafir]

@avikivity v3:

• Allow seastar::join_memory_prefault only on shard 0, enforce via assertion and document. Keep run_in_background() which is needed.
• Remove redundant warning.
• If memory prefault is not initialized, instead of checking inside seastar::join_memory_prefault, submit dummy completion immediately in smp::configure.