Scalable Active Message Multicast - #465
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #465 +/- ##
==========================================
+ Coverage 30.41% 34.94% +4.53%
==========================================
Files 199 202 +3
Lines 41281 44162 +2881
Branches 14942 15266 +324
==========================================
+ Hits 12554 15433 +2879
+ Misses 28272 27383 -889
- Partials 455 1346 +891 ☔ View full report in Codecov by Harness. |
|
I do not think we should force every network module to use tree based broadcast. IMHO, we could provide an abstraction function ( |
To be clear, at the moment, none of our networking layers have native hardware/driver level support for multi-cast active messages which is why the tree broadcast logic all lives on the "client" side of the |
2 participants
Motivation
ActiveMessage(NodeSet) was multicast in name only. Each network backend expanded the target set at the source and issued one send per target:
for(NodeID tgt : targets) { ... internal->commit_message(...); }
So a message to N peers cost the originating node N sends, and the target list travelled the wire as a raw node set. This adds a real multicast facility — bounded-radix forwarding over ordinary unicast, with an adaptive target encoding — and deletes the per-target source loops from all four backends.
What changed
A standalone target-set codec — src/realm/multicast.{h,cc}
MulticastTargetSet stores sorted, disjoint, non-adjacent runs, so a range covering thousands of nodes costs one entry and partition() slices it without ever expanding to individual IDs. EncodedMulticastTargets picks among eight wire forms — EMPTY, SINGLE, SMALL_INLINE, RANGES, DELTA_LIST, BITMAP, ALL_NODES, ALL_EXCEPT — by computing the actual serialized byte size of every candidate and taking the minimum, not by a density heuristic. Decoding validates cardinality against both the remaining payload and the configured node count before allocating or looping, plus node bounds, range overflow, canonical ordering and overlong varints; malformed input returns a status rather than aborting, so it stays unit-testable.
This layer deliberately has no dependency on activemsg.h, which is what lets its tests run with no network, no runtime and no message table.
Forwarding, folded into the active-message layer — activemsg.{h,inl,cc}
The origin removes itself from the target set, partitions the remainder into at most R near-equal slices, and sends one envelope to the first node of each. A relay validates it is in its own slice, removes itself, then enqueues child envelopes before invoking the local handler — forward-before-deliver, which matters for messages like runtime shutdown whose handler stops progress. Final delivery presents origin_node as the handler's sender, not the last relay.
Depth is O(log_R M) with O(M) total edges. Handler redispatch reuses the existing ActiveMessageHandlerTable / IncomingMessageManager machinery rather than duplicating signature detection, and large envelopes ride the existing fragmentation path on every hop.
Optional aggregate remote completion uses a transient acknowledgement tree: each relay holds its parent and outstanding-child count, acks once, and immediately reclaims. Fire-and-forget multicasts carry no ack metadata and create no state, and no reusable target plan survives a completed multicast.
Call sites migrated: runtime shutdown (runtime_impl.cc), metadata invalidation (metadata.cc), instance requests (inst_impl.cc), CUDA and HIP IPC.
Removed: ActiveMessage(NodeSet), Network::create_active_message_impl(NodeSet, …), the NetworkModule NodeSet virtuals, the NodeSet recommended_max_payload overloads, and the backend multicast constructors and source loops — 13 files, +51 / −669. Realm::NodeSet itself is unchanged; only the multicast-by-NodeSet path is gone.
Testing
101 new unit tests. Every encoding is forced with an explicit kind assertion, and a shared helper independently recomputes the minimum across all eight candidates to confirm the encoder actually picked it. Coverage includes malformed and fuzzed payloads (verified against a guard page to catch out-of-bounds reads), NodeID 0 and max, partition balance/disjointness/exact-union across radices 1–64, forward-before-deliver ordering, origin-sender preservation across multiple hops, fragmented payloads, and completion firing exactly once.