Output Regions Need Explicit Free Method When Assuming Ownership of Naked Realm Instances

[Jacobfaib]

Is your feature request related to a problem? Please describe.

I am giving Realm access to some memory I have allocated. I want to be able to tell Realm how to delete this memory later on when it is no longer needed. Currently, I need to separately track when the memory is no longer used (somehow) and call the deleter myself.

Describe the solution you'd like

Realm knows best when to delete the memory I have given it. It should know how to do that. I would like to pass a std::function<void(void*)> to Realm::ExternalMemoryResource.

For example:

auto* memory = my_special_cuda_malloc(size_bytes, ...);

auto resource = Realm::ExternalMemoryResource{
  /* mem */ memory, 
  /* size */ size_bytes, 
  /* deleter */ &my_special_cuda_free
};

Describe alternatives you've considered

Manually tracking the memory lifetime myself and deleting it. This is very cumbersome and bugprone.

[lightsighter]

Currently, I need to separately track when the memory is no longer used (somehow) and call the deleter myself.

You have to do this anyway when you delete the instance associated with an ExternalMemoryResource. Since you're already required to manually manage the lifetime of the instance then it shouldn't be that hard to also free up the memory associated with the instance at the same time (unless you want to keep the memory around longer).

To take a step back, it's worth noting that ExternalInstanceResource descriptors and their derivations like ExternalMemoryResource are not designed for Realm to take ownership of the external resources. They are provided as a way to create an instance backed by some external memory such that Realm can understand the layout of that external memory and do Realm operations like copies/fills/depparts to/from the external memory. When the client creates an instance from an ExternalInstanceResource, it's still the client's responsibility to ensure that the external resource remains alive as long as the instance is in use. However, the opposite direction is not necessarily true: it's fine to delete an instance you create from an ExternalInstanceResource long before you free the external resource.

@Jacobfaib is it more pertinent to have this conversation in a Legion context where Legion does let you pass in an explicit free function when you pass in an ExternalInstanceResource as a return value for a task, e.g.:
https://gitlab.com/StanfordLegion/legion/-/blob/master/runtime/legion.h?ref_type=heads#L10277
Or are you using Realm's ExternalInstanceResource in some other context?

[Jacobfaib]

To take a step back, it's worth noting that ExternalInstanceResource descriptors and their derivations like ExternalMemoryResource are not designed for Realm to take ownership of the external resources.

I think this design is worth reconsidering, at least for external memory resources, because...

it's still the client's responsibility to ensure that the external resource remains alive as long as the instance is in use.

It's not easy for the user to figure out exactly what the lifetimes are once it leaves their hands, for example when it disappears into some other library (such as Legion). Now I have to know about how Legion manages the lifetime of that resource in order to keep it alive long enough. The more layers there are, the leakier this abstraction becomes.

I am already telling Realm a great deal about the memory, so it's not too far a leap to also tell Realm how to dispose of the memory. This would make handling of this memory much simpler across the stack.

is it more pertinent to have this conversation in a Legion context where Legion does let you pass in an explicit free function when you pass in an ExternalInstanceResource

Sure, yeah, that would solve my immediate problem. But I still maintain that Realm should be able to manage this memory resource on its own.

[apryakhin]

Sure, yeah, that would solve my immediate problem. But I still maintain that Realm should be able to manage this memory resource on its own.

Realm isn't owning this resource, it can't simply know who is using the underlying buffer. For example, you may need it to live past the instance lifetime or reuse for another instance. What if one created another instance out of this external memory resource...do you want to realm somehow to understand and reference count that?

It's not easy for the user to figure out exactly what the lifetimes are once it leaves their hands

Why? The application has full control of this external resource, you know when you need to delete a corresponding external instance and you know when this deletion is complete...can't you just simply clean up the external resource as well as this point.

[Jacobfaib]

Why? The application has full control of this external resource, you know when you need to delete a corresponding external instance and you know when this deletion is complete...can't you just simply clean up the external resource as well as this point.

This is a circular argument: "Why don't you know the lifetime, you can just figure it out". My point is I don't want to know :).

If I create a Realm instance, and I pass this instance to Legion, I now need to know exactly what Legion will do with that Realm instance, and keep the memory alive for that duration. Or I need to punt this responsibility up to my caller, who has no idea what a Realm::ExternalResource even is and why they need to work around it. This isn't scalable.

[lightsighter]

It's not easy for the user to figure out exactly what the lifetimes are once it leaves their hands, for example when it disappears into some other library (such as Legion). Now I have to know about how Legion manages the lifetime of that resource in order to keep it alive long enough.

If an abstraction built on top of Realm agrees to take ownership of an (external) instance then it should also agree to take ownership of the external memory resource. However, the opposite should also be possible. Either way, I think both can be accomplished on top of the existing interface without needing to change Realm. If there are cases where you think Legion isn't doing a good job of managing these lifetimes then we can have that discussion in the context of Legion.

The more layers there are, the leakier this abstraction becomes.
I am already telling Realm a great deal about the memory, so it's not too far a leap to also tell Realm how to dispose of the memory. This would make handling of this memory much simpler across the stack.

For memory that Realm manages, this should absolutely be true and it is. However, we're talking about external memory resources here which by definition are outside of Realm's control and therefore have uses that will dictate the lifetime of those resources beyond the scope of Realm. (Otherwise you could have just used normal Realm instances.) Therefore it seems unreasonable for Realm to have to take ownership of some resource which by definition is beyond its scope of understanding how to manage the lifetime for. This is further reflected in the design with explicit instance deletions. You can't escape the problem of lifetimes as you're still going to have to explicitly say when to delete the external instance that is made from an external resource. So really all we're discussing here is adding a convenience method to free memory after the Realm client has done all the hard work to determine the lifetime of the associated instance and explicitly deleted it. I think the value proposition on that is quite low considering the user is already doing all the hard work anyway to figure out when to delete the instance, and it opens Realm up to a class of bugs (crashes in callback functions) which I think are better avoided.

My point is I don't want to know :)

You have to know though. Even if you get the convenience wrapper you're asking for deleting external resources when an associated instance is deleted, you still have to know the lifetime for the instance itself. The problem of lifetimes does not go away. The only way to fix this would be to add a distributed garbage collector on instance handles across the whole machine (which we're not going to do for a multitude of reasons).

If I create a Realm instance, and I pass this instance to Legion, I now need to know exactly what Legion will do with that Realm instance, and keep the memory alive for that duration.

If we want to have this conversation in the context of Legion I think that is more reasonable. I think you'll also find that Legion offers both ways of having Legion take ownership of external resources (as well as not) for exactly the reasons that you describe. I'm open to discussing places where Legion is not providing the semantics that you want, but that is well beyond the scope of Realm.

[lightsighter]

As resolved in the meeting, what really needs to happen here is that Legion needs to provide a way for taking an explicit free function for external resources of naked Realm instances passed into output regions the same way that it does for futures when it takes ownership of such things. Therefore I'm going to move this issue over to the Legion repo and rename it appropriately.