Skip to content

Phoebe/parallelism test/rayon par iter only #2552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Phoebe/parallelism test/rayon par iter only #2552

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cae2628
Rip Rayon out of reducer execution and query evaluation
gefjon Apr 2, 2025
9fe73d3
Merge remote-tracking branch 'origin/master' into phoebe/parallelism-…
gefjon Apr 4, 2025
f28dc30
Second experiment: use `par_iter`, but not `scope` or `spawn`
gefjon Apr 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions crates/core/src/host/wasm_common/module_host_actor.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,9 +462,9 @@ impl<T: WasmInstance> WasmModuleInstance<T> {
)
.entered();

// run the call_reducer call in rayon. it's important that we don't acquire a lock inside a rayon task,
// as that can lead to deadlock.
let (mut tx, result) = rayon::scope(|_| tx_slot.set(tx, || self.instance.call_reducer(op, budget)));
// FOR BENCHMARKING: Just run the reducer on whatever thread we're already on,
// instead of bouncing to a Rayon thread.
let (mut tx, result) = tx_slot.set(tx, || self.instance.call_reducer(op, budget));

let ExecuteResult {
energy,
Expand Down
7 changes: 4 additions & 3 deletions crates/core/src/subscription/module_subscription_manager.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -463,8 +463,9 @@ impl SubscriptionManager {

let tables = &event.status.database_update().unwrap().tables;

// Put the main work on a rayon compute thread.
rayon::scope(|_| {
// FOR TESTING: Don't put the main work on a rayon compute thread.
// Do still use `par_iter` to parallelize.
{
let span = tracing::info_span!("eval_incr").entered();
let (updates, errs, metrics) = tables
.iter()
Expand Down Expand Up @@ -685,7 +686,7 @@ impl SubscriptionManager {
);
}
}
})
}
}
}

Expand Down
1 change: 0 additions & 1 deletion crates/core/src/subscription/subscription.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,6 @@ impl ExecutionSet {
slow_query_threshold: Option<Duration>,
compression: Compression,
) -> ws::DatabaseUpdate<F> {
// evaluate each of the execution units in this ExecutionSet in parallel
let tables = self
.exec_units
// if you need eval to run single-threaded for debugging, change this to .iter()
Expand Down
Loading