Run adaptive localization in parallel - #228
Merged
Merged
Conversation
xjules
reviewed
Jun 16, 2025
JHolba
reviewed
Jul 2, 2025
xjules
reviewed
Aug 4, 2025
| return corr_XY | ||
|
|
||
| @staticmethod | ||
| def _update_single_parameter( |
Contributor
There was a problem hiding this comment.
_update_single_parameter_group?
xjules
reviewed
Aug 4, 2025
|
|
||
| @staticmethod | ||
| def _update_single_parameter( | ||
| param_num: int, |
Contributor
There was a problem hiding this comment.
Why we need to provide param_num?
Contributor
There was a problem hiding this comment.
It seems that Parallel(n_jobs=n_jobs, return_as="list") (default setting) preserves the order of the outputs.
xjules
reviewed
Aug 4, 2025
xjules
reviewed
Aug 4, 2025
xjules
approved these changes
Aug 4, 2025
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Using
joblibto loop over parameters in parallel when doing adaptive localization.Added the test
test_parallelization_runtime_comparisonfor benchmarking which uses the fixturelarge_linear_problemto set-up a problem with lots of parameters and observations.Synthetic tests
Below is an example run. Results seem promising but more testing is needed.
num_parameters = 100000
num_observations = 200
num_ensemble = 200
--- Benchmark Summary ---
✅ Numerical results are identical.
🚀 Speedup factor: 4.84x
Drogon
Komodo
testing37 min:Komodo
testingwith parallel version 19 min:Almost 2x speed-up. Note that we in the synthetic test force every parameter to be updated by setting the threshold to 0, while drogon is run with a threshold of 0.3. There might be a significant number of parameters that do not get updated, but I'm not sure.
Memory usage
From docs of joblib.Parallel