fix(pm): bound install pipeline task concurrency#2849
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to limit parallel I/O operations during package installation and pipeline processing by utilizing tokio::task::JoinSet and a new parallel_io_limit utility. This change aims to reduce scheduler churn and manage resource usage more effectively. A review comment identifies that the join_one helper in the pipeline worker ignores task results, potentially swallowing panics, and suggests propagating them to ensure unrecoverable bugs are properly surfaced.
| async fn join_one(tasks: &mut tokio::task::JoinSet<()>) { | ||
| let _ = tasks.join_next().await; | ||
| } |
There was a problem hiding this comment.
The join_one helper currently ignores the result of join_next(), which means any panics occurring in the pipeline subtasks will be silently swallowed. According to the general rules, panics should be treated as unrecoverable bugs and not be ignored. It is recommended to check if the task panicked and propagate it using resume_unwind to ensure that bugs in the pipeline workers are surfaced.
async fn join_one(tasks: &mut tokio::task::JoinSet<()>) {
if let Some(Err(e)) = tasks.join_next().await {
if e.is_panic() {
std::panic::resume_unwind(e.into_panic());
}
}
}References
- Do not implement recovery logic for panics. Panics should be treated as unrecoverable bugs that need to be fixed, not as transient, recoverable errors.
|
Closing as stale: this draft is a one-off agent experiment from 2026-04-27 with no follow-up, and overlaps with sibling PRs exploring the same optimization. Reopen if revisited. |
Summary
Validation
Notes