Skip to content

Conversation

@j6k4m8
Copy link
Member

@j6k4m8 j6k4m8 commented Jul 27, 2020

By default, uses n_cpu parallel threads to upload. Now symmetric with get_cutout which has been parallel since v1.0.0.

Fixes #62

@j6k4m8 j6k4m8 requested review from dxenes1, movestill and wrgr July 27, 2020 18:19
@j6k4m8
Copy link
Member Author

j6k4m8 commented Mar 21, 2021

There appear to be some nasties w.r.t numpy shared memory when slicing and uploading a large array. I think the default behavior might be to copy the array across each thread, which is no good for RAM. Was really hoping to avoid doing a deeper dive here but looks like we might need to... I'm also seeing a bunch of 500 errors in the test logs which is weird.

@dxenes1 I wonder if you encountered the same on the download side?

movestill
movestill previously approved these changes Mar 21, 2021
@dxenes1
Copy link
Contributor

dxenes1 commented Mar 22, 2021

There appear to be some nasties w.r.t numpy shared memory when slicing and uploading a large array. I think the default behavior might be to copy the array across each thread, which is no good for RAM. Was really hoping to avoid doing a deeper dive here but looks like we might need to... I'm also seeing a bunch of 500 errors in the test logs which is weird.

@dxenes1 I wonder if you encountered the same on the download side?

Didn't see anything with download, maybe since I'm downloading raw bytearrays and then constructing the numpy array at the end i don't run into the same issue when downloading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelize BossRemote#create_cutout

4 participants