-
Notifications
You must be signed in to change notification settings - Fork 15
Add parallelism to BossRemote#create_cutout #64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
There appear to be some nasties w.r.t numpy shared memory when slicing and uploading a large array. I think the default behavior might be to copy the array across each thread, which is no good for RAM. Was really hoping to avoid doing a deeper dive here but looks like we might need to... I'm also seeing a bunch of 500 errors in the test logs which is weird. @dxenes1 I wonder if you encountered the same on the download side? |
Didn't see anything with download, maybe since I'm downloading raw bytearrays and then constructing the numpy array at the end i don't run into the same issue when downloading. |
By default, uses
n_cpuparallel threads to upload. Now symmetric withget_cutoutwhich has been parallel since v1.0.0.Fixes #62