Production Quality Compiled Subgraph Implementation

[rohany]

An original prototype of my subgraph implementation is in this pull request: #391.

Based on several discussions, I started the process of extracting the pull request into several pieces, those starting PRs are here: #394, #416, #417.

The prototype implementation performs the most acceleration for subgraphs that have operations that are planned to run on the node that created the subgraph. The implementation:

• Caches copy plans and XD's for fast copy replays
• Removes event-based dependence tracking whenever possible
• Allows for host-side run ahead of device-side operations (GPU tasks and copies)

The implementation does this by registering a separate "subgraph executor" that takes over Realm's standard processor threads, executing dedicated subgraph work until a subgraph completes.

The initial batch of new PRs start rebuilding this infrastructure on the latest realm by starting with simple CPU-only subgraphs. I had several pieces of future work planned to gain feature parity with the prototype implementation:

• support external pre- and post-conditions in the compiled subgraph
• support escaping from non-compiled components into the standard interpreted subgraph
• ensuring that subgraph tasks can wait on events within their tasks, which means the changes to the thread scheduler need to enable preemption.
• making the XD infrastructure resettable
• adding hooks to the XD infrastructure to enable being called from the subgraph execution framework
• support extraction of "current finish event" from tasks that are executing with the subgraph
• add an interface to realm tasks to promise that GPU tasks are stream-ordered
• negotiate dependencies between GPU tasks directly through CUDA events instead of Realm events. This would also need to be done for copies that are serviced by the device.

Future work can investigate extending these optimizations to multi-node settings, using advanced features like NVL72 etc for low-latency cross-node communication, or plan better cross-node communication patterns.