Skip to content

fix(cutile): synchronize stream in to_host_vec before dropping owned tensor - #278

Open
emersonbusson wants to merge 1 commit into
NVlabs:mainfrom
emersonbusson:fix/to-host-vec-stream-ordering
Open

emersonbusson wants to merge 1 commit into
NVlabs:mainfrom
emersonbusson:fix/to-host-vec-stream-ordering

Conversation

@emersonbusson

Copy link
Copy Markdown
Contributor

Summary

Fixes #252.

Synchronizes the execution stream in CopyDeviceToHostVec::execute before dropping self.tensor, eliminating a stream-ordered race and use-after-free between the in-flight DtoH memcpy and asynchronous deallocation on the deallocator_stream.

Root Cause

When copying an owned tensor to host memory via tensor.to_host_vec().sync_on(&stream):

  1. CopyDeviceToHostVec holds ownership of the source tensor via self.tensor: Arc<Tensor<T>>.
  2. In CopyDeviceToHostVec::execute, memcpy_dtoh_async is enqueued on ctx.get_cuda_stream().
  3. execute(self, &ctx) consumes self by value, meaning self (and its inner self.tensor) is dropped at the return of execute.
  4. Dropping the owned tensor triggers DeviceBuffer::drop, which immediately enqueues cuMemFreeAsync on the thread's deallocator_stream.
  5. Because deallocator_stream does not establish an event wait on ctx.get_cuda_stream(), the free operation on deallocator_stream races with the in-flight copy on ctx.get_cuda_stream(), triggering a stream-ordered use-after-free flagged by Compute-Sanitizer (as reported in [BUG]: Possible use after free due to deallocation being queued on a different stream before sync_on is called #252).

Fix

By explicitly calling ctx.get_cuda_stream().synchronize() inside CopyDeviceToHostVec::execute before self.tensor is dropped:

  • The device read on ctx.get_cuda_stream() is guaranteed to be 100% complete before any deallocation can be submitted to the deallocator_stream.
  • Ensures the safety contract (SAFETY: ... returns only once the copy has completed, so all size elements are initialized here) holds deterministically across all stream configurations.

Validation

@copy-pr-bot

copy-pr-bot Bot commented Sep 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Possible use after free due to deallocation being queued on a different stream before sync_on is called

1 participant