Skip to content

RF: Use safe resizing for ArraySequence extension #724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 12, 2019

Conversation

effigies
Copy link
Member

@effigies effigies commented Feb 8, 2019

In #722, ArraySequence tests generate ndarray.resize failures similar to #719, when running ArraySequence._resize_data_to(). Apparently the combination of Python 3.7 and coverage produce a ghost reference.

In this case, it's not clear that it's being called in a safe way such that we should skip reference checks. I added a test that shows that pulling a slice for use is sufficient to create a reference that causes extension to fail. This seems to be a case where we don't want to skip the refcheck, as a resize that increases the array size will typically release the original memory, corrupting data that users would expect to be unchanged.

The obvious solution is to switch to np.resize here, but np.resize returns views on an inaccessible object, so they don't have the OWNDATA flag, which means we can no longer use ndarray.resize in ArraySequence.shrink_data. A possible remedy is using np.resize().copy(), which will give us back control, and count on the garbage collector to take care of things, but until it does, we'll be seeing 3 copies of a single data array. This is in contrast to ndarray.resize, which uses realloc under the hood, so is pretty close to 1 copy.

Here I add a _safe_resize helper function. Because we can guarantee the second ndarray.resize will work, the intermediate a will be deallocated by realloc, not the garbage collector. This will also continue to have the current memory profile, if nobody ever tries to slice the data and then extend (which seems to be the case, as we haven't had any bug reports).

@MarcCote If you get some time, would you mind reviewing this? Or anyone else who feels comfortable.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.005%) to 91.844% when pulling 989e6bb on effigies:test/array_seq_extension into 7927c2d on nipy:master.

1 similar comment
@coveralls
Copy link

Coverage Status

Coverage increased (+0.005%) to 91.844% when pulling 989e6bb on effigies:test/array_seq_extension into 7927c2d on nipy:master.

Copy link
Contributor

@MarcCote MarcCote left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow! Thanks for sorting this out @effigies. I agree with you that extending an ArraySequence after making a slice must not happen that often.

@effigies
Copy link
Member Author

Cool. Thanks for the review.

@effigies effigies merged commit 5537931 into nipy:master Feb 12, 2019
@effigies effigies deleted the test/array_seq_extension branch February 12, 2019 02:15
@effigies effigies added this to the 2.4.0 milestone Mar 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants