Skip to content

Commit da94398

Browse files
committed
fix(event): own graph guard-condition registrations instead of raw pointers
Moving the guard-condition trigger out of the registry lock was necessary -- the trigger is rmw code that re-enters hiroz -- but it turned a deadlock into a use-after-free. `trigger_graph_change` snapshotted `Vec<usize>` of raw pointers, released the lock, then dereferenced them. The lock was the only thing serialising that against teardown: `rmw_destroy_node` calls `unregister_graph_guard_condition` and then immediately `rmw_destroy_guard_condition`, which frees the handle. Previously `unregister` blocked until triggering finished, so the free could not land mid-trigger. Afterwards it could, and the trigger wrote through freed memory. Concurrent triggers also aliased `&mut GuardConditionImpl`. Make registrations owned. hiroz gains a `GraphGuardCondition` trait and stores `Arc<dyn GraphGuardCondition>`; the snapshot clones `Arc`s, so an in-flight trigger keeps its target alive no matter what teardown does. `unregister` is by `Arc::ptr_eq` and no longer implies "no trigger is running" -- it does not need to, because the survivor keeps the object alive. rmw-zenoh-rs splits `GuardConditionImpl` into a C-side handle and an `Arc<GuardConditionState>` holding the notifier and an `AtomicBool`. The node registers a clone of that state and keeps one itself for unregistration. `triggered` becomes atomic because triggering no longer happens under any lock. The process-wide `set_guard_condition_trigger` indirection is gone -- each registration now carries its own behaviour -- so `GraphGuardConditionTrigger` is removed. Covered by a unit test pinning the ownership contract: the registry keeps the value alive after the registrant drops its handle, and releases it on unregister. That property is what makes the destroy-during-trigger race harmless, and unlike the race itself it is deterministic to assert.
1 parent af5d0b3 commit da94398

4 files changed

Lines changed: 196 additions & 60 deletions

File tree

crates/hiroz/src/event.rs

Lines changed: 104 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -177,19 +177,37 @@ impl EventsManager {
177177
}
178178
}
179179

180-
// Callback type for triggering graph guard conditions.
181-
//
182-
// `Arc` for the same reason as [`EventCallback`]: it is cloned out of its
183-
// `Mutex` before being called, so guard-condition triggers never run with the
184-
// graph-event registry locked.
185-
pub type GraphGuardConditionTrigger = Arc<dyn Fn(*mut std::ffi::c_void) + Send + Sync>;
180+
/// A graph guard condition this manager may trigger on a graph change.
181+
///
182+
/// Registrations are **owned**, not raw pointers, and that is the whole point.
183+
/// Triggering happens after the registry lock is released — it has to, because
184+
/// the trigger is rmw-side code that re-enters hiroz. But a raw pointer cloned
185+
/// out of the lock is only valid while something guarantees the target outlives
186+
/// the call, and nothing did: `rmw_destroy_node` unregisters and then
187+
/// immediately frees the guard condition, so a destroy landing between the
188+
/// snapshot and the call left the trigger dereferencing freed memory.
189+
///
190+
/// Holding an `Arc` for the duration of the call closes that window without
191+
/// reintroducing the lock: the implementation's state stays alive as long as
192+
/// this manager holds a reference, even if the C-side handle is destroyed
193+
/// concurrently. Implementors must therefore keep [`trigger`] valid after the
194+
/// owning C object is gone — the natural shape is state behind its own `Arc`,
195+
/// with the C handle holding one reference and this registry another.
196+
///
197+
/// [`trigger`]: GraphGuardCondition::trigger
198+
pub trait GraphGuardCondition: Send + Sync {
199+
/// Wake whatever is waiting on this guard condition.
200+
///
201+
/// Called with no hiroz lock held, possibly concurrently, and possibly
202+
/// after the corresponding C handle has been destroyed.
203+
fn trigger(&self);
204+
}
186205

187206
// GraphCache event integration
188207
pub struct GraphEventManager {
189208
event_callbacks: TrackedMutex<HashMap<GidArray, HashMap<ZenohEventType, EventCallback>>>,
190209
entity_topics: TrackedMutex<HashMap<GidArray, String>>, // Topic name per registered entity
191-
graph_guard_conditions: TrackedMutex<Vec<usize>>, // Pointers as usize for Send
192-
trigger_guard_condition: TrackedMutex<Option<GraphGuardConditionTrigger>>,
210+
graph_guard_conditions: TrackedMutex<Vec<Arc<dyn GraphGuardCondition>>>,
193211
}
194212

195213
impl Default for GraphEventManager {
@@ -204,14 +222,9 @@ impl GraphEventManager {
204222
event_callbacks: TrackedMutex::new(HashMap::new()),
205223
entity_topics: TrackedMutex::new(HashMap::new()),
206224
graph_guard_conditions: TrackedMutex::new(Vec::new()),
207-
trigger_guard_condition: TrackedMutex::new(None),
208225
}
209226
}
210227

211-
pub fn set_guard_condition_trigger(&self, trigger: GraphGuardConditionTrigger) {
212-
*self.trigger_guard_condition.lock().unwrap() = Some(trigger);
213-
}
214-
215228
pub fn register_event_callback<F>(
216229
&self,
217230
entity_gid: GidArray,
@@ -244,15 +257,27 @@ impl GraphEventManager {
244257
topics.remove(entity_gid);
245258
}
246259

247-
pub fn register_graph_guard_condition(&self, guard_condition: *mut std::ffi::c_void) {
260+
/// Register a guard condition to be triggered on every graph change.
261+
///
262+
/// The manager keeps the `Arc` alive for as long as it is registered, and
263+
/// for the duration of any trigger already in flight — see
264+
/// [`GraphGuardCondition`] for why that ownership is load-bearing.
265+
pub fn register_graph_guard_condition(&self, guard_condition: Arc<dyn GraphGuardCondition>) {
248266
let mut conditions = self.graph_guard_conditions.lock().unwrap();
249-
conditions.push(guard_condition as usize);
267+
conditions.push(guard_condition);
250268
}
251269

252-
pub fn unregister_graph_guard_condition(&self, guard_condition: *mut std::ffi::c_void) {
270+
/// Stop triggering `guard_condition`.
271+
///
272+
/// Identity is `Arc::ptr_eq`, so the caller must pass the same allocation it
273+
/// registered. Returning does **not** mean no trigger is in flight: a
274+
/// concurrent [`Self::trigger_graph_change`] may already hold its own clone
275+
/// and be calling into it. That is exactly why the registration is owned —
276+
/// the in-flight call keeps the target alive, so a caller that frees its own
277+
/// handle immediately after this returns is still safe.
278+
pub fn unregister_graph_guard_condition(&self, guard_condition: &Arc<dyn GraphGuardCondition>) {
253279
let mut conditions = self.graph_guard_conditions.lock().unwrap();
254-
let gc_usize = guard_condition as usize;
255-
conditions.retain(|&gc| gc != gc_usize);
280+
conditions.retain(|gc| !Arc::ptr_eq(gc, guard_condition));
256281
}
257282

258283
pub fn trigger_event(&self, entity_gid: &GidArray, event_type: ZenohEventType, change: i32) {
@@ -309,15 +334,16 @@ impl GraphEventManager {
309334
let change = if appeared { 1 } else { -1 };
310335

311336
// Trigger graph guard conditions for ALL graph changes (local and remote).
312-
// Snapshot the trigger and the pointer list, then release both locks before
313-
// calling out — the trigger is rmw-side code and may re-enter.
314-
let trigger = self.trigger_guard_condition.lock().unwrap().clone();
315-
if let Some(trigger) = trigger {
316-
let guard_conditions = self.graph_guard_conditions.lock().unwrap().clone();
317-
for gc_usize in guard_conditions {
318-
let gc = gc_usize as *mut std::ffi::c_void;
319-
trigger(gc);
320-
}
337+
//
338+
// Snapshot, release the lock, then call — the trigger is rmw-side code
339+
// and may re-enter this manager, so it must not run under the guard.
340+
// The snapshot clones `Arc`s rather than raw pointers, which is what
341+
// makes releasing the lock safe: a concurrent `rmw_destroy_node` can
342+
// unregister and free its C handle here, and each in-flight trigger
343+
// still holds the target alive until it returns.
344+
let guard_conditions = self.graph_guard_conditions.lock().unwrap().clone();
345+
for gc in guard_conditions {
346+
gc.trigger();
321347
}
322348

323349
// Determine which event type based on entity kind
@@ -696,4 +722,55 @@ mod tests {
696722
.update_event_status(ZenohEventType::LivelinessChanged, 3);
697723
assert_eq!(*fired.lock().unwrap(), 3);
698724
}
725+
726+
/// The graph-guard-condition registry must **own** what it registers.
727+
///
728+
/// This is the invariant that makes triggering outside the lock safe.
729+
/// `trigger_graph_change` snapshots the registrations, releases the lock,
730+
/// and only then calls them; meanwhile `rmw_destroy_node` may unregister
731+
/// and free its C handle. When the registry held raw pointers, that window
732+
/// was a use-after-free. Holding an `Arc` closes it — an in-flight trigger
733+
/// keeps the target alive regardless of what the registrant does.
734+
///
735+
/// The test pins the ownership half of that contract, which is the part
736+
/// that is deterministic: the registry keeps the value alive after the
737+
/// registrant drops its handle, and releases it on unregister. It does not
738+
/// attempt to schedule the destroy-during-trigger race itself — that would
739+
/// be a timing test, and the ownership property is what makes the race
740+
/// harmless in the first place.
741+
#[test]
742+
fn graph_guard_condition_registration_is_owned_by_the_manager() {
743+
use std::sync::atomic::{AtomicUsize, Ordering};
744+
745+
struct CountingGc(Arc<AtomicUsize>);
746+
impl GraphGuardCondition for CountingGc {
747+
fn trigger(&self) {
748+
self.0.fetch_add(1, Ordering::SeqCst);
749+
}
750+
}
751+
752+
let mgr = GraphEventManager::new();
753+
let hits = Arc::new(AtomicUsize::new(0));
754+
let gc: Arc<dyn GraphGuardCondition> = Arc::new(CountingGc(hits.clone()));
755+
let weak = Arc::downgrade(&gc);
756+
757+
mgr.register_graph_guard_condition(gc.clone());
758+
drop(gc);
759+
let held = weak.upgrade().expect(
760+
"the manager must keep the registration alive after the registrant drops its handle; \
761+
otherwise a trigger issued outside the lock dereferences freed memory",
762+
);
763+
764+
// Still reachable and callable through the registry's own reference.
765+
held.trigger();
766+
assert_eq!(hits.load(Ordering::SeqCst), 1);
767+
768+
mgr.unregister_graph_guard_condition(&held);
769+
drop(held);
770+
assert!(
771+
weak.upgrade().is_none(),
772+
"unregister must release the registry's reference, or registrations leak for the \
773+
life of the process",
774+
);
775+
}
699776
}

crates/rmw-zenoh-rs/src/context.rs

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -95,18 +95,6 @@ impl ContextImpl {
9595
.build()
9696
.map_err(|e| format!("Failed to create ZContext: {}", e))?;
9797

98-
// Set up the guard condition trigger function for graph events
99-
// This allows graph changes to trigger RMW guard conditions
100-
let trigger_fn: hiroz::event::GraphGuardConditionTrigger = Arc::new(|gc_ptr| {
101-
crate::guard_condition::rmw_trigger_guard_condition(
102-
gc_ptr as *const crate::ros::rmw_guard_condition_t,
103-
);
104-
});
105-
zcontext
106-
.graph()
107-
.event_manager
108-
.set_guard_condition_trigger(trigger_fn);
109-
11098
Ok(Self {
11199
zcontext: Arc::new(zcontext),
112100
enclave,

crates/rmw-zenoh-rs/src/guard_condition.rs

Lines changed: 58 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,30 +3,76 @@ use crate::ros::*;
33
use crate::traits::*;
44
use crate::utils::Notifier;
55
use std::sync::Arc;
6+
use std::sync::atomic::{AtomicBool, Ordering};
7+
8+
/// The triggerable state of a guard condition, separated from the C handle.
9+
///
10+
/// This lives behind an `Arc` so it can outlive `rmw_destroy_guard_condition`.
11+
/// hiroz's graph-event manager registers a clone (as
12+
/// `Arc<dyn GraphGuardCondition>`) and triggers it **after** dropping its
13+
/// registry lock; without shared ownership, a `rmw_destroy_node` racing that
14+
/// window would free the target and the trigger would write through dangling
15+
/// memory. The C handle holds one reference and the registry another, so
16+
/// whichever outlives the other, the state is valid for the whole call.
17+
///
18+
/// `triggered` is atomic because triggering no longer happens under any lock:
19+
/// a graph-event thread can set it while `rmw_wait` reads it.
20+
#[derive(Debug, Default)]
21+
pub struct GuardConditionState {
22+
pub(crate) notifier: Option<Arc<Notifier>>,
23+
pub(crate) triggered: AtomicBool,
24+
}
25+
26+
impl GuardConditionState {
27+
pub(crate) fn fire(&self) -> Result<(), ()> {
28+
let notifier = self.notifier.as_ref().ok_or(())?;
29+
self.triggered.store(true, Ordering::SeqCst);
30+
notifier.notify_all();
31+
Ok(())
32+
}
33+
34+
pub(crate) fn reset(&self) {
35+
self.triggered.store(false, Ordering::SeqCst);
36+
}
37+
38+
pub(crate) fn is_triggered(&self) -> bool {
39+
self.triggered.load(Ordering::SeqCst)
40+
}
41+
}
42+
43+
impl hiroz::event::GraphGuardCondition for GuardConditionState {
44+
fn trigger(&self) {
45+
// A guard condition with no notifier cannot wake anyone; that is not an
46+
// error worth propagating across the registry.
47+
let _ = self.fire();
48+
}
49+
}
650

751
/// Guard condition implementation for RMW
852
#[derive(Debug, Default)]
953
pub struct GuardConditionImpl {
10-
pub(crate) notifier: Option<Arc<Notifier>>,
11-
pub(crate) triggered: bool,
54+
pub(crate) state: Arc<GuardConditionState>,
1255
}
1356

1457
impl GuardConditionImpl {
1558
pub(crate) fn trigger(&mut self) -> Result<(), ()> {
16-
let notifier = self.notifier.as_ref().ok_or(())?;
17-
self.triggered = true;
18-
notifier.notify_all();
19-
Ok(())
59+
self.state.fire()
2060
}
2161

2262
pub fn reset(&mut self) {
23-
self.triggered = false;
63+
self.state.reset();
64+
}
65+
66+
/// A shared handle to this guard condition's state, for registration with
67+
/// hiroz's graph-event manager.
68+
pub(crate) fn share_state(&self) -> Arc<GuardConditionState> {
69+
self.state.clone()
2470
}
2571
}
2672

2773
impl crate::traits::Waitable for GuardConditionImpl {
2874
fn is_ready(&self) -> bool {
29-
self.triggered
75+
self.state.is_triggered()
3076
}
3177
}
3278

@@ -52,8 +98,10 @@ pub extern "C" fn rmw_create_guard_condition(
5298

5399
let notifier = Some(context_impl.share_notifier());
54100
let gc_impl = GuardConditionImpl {
55-
notifier,
56-
triggered: false,
101+
state: Arc::new(GuardConditionState {
102+
notifier,
103+
triggered: AtomicBool::new(false),
104+
}),
57105
};
58106
let gc = Box::new(rmw_guard_condition_t {
59107
implementation_identifier: crate::RMW_ZENOH_IDENTIFIER.as_ptr() as *const _,

crates/rmw-zenoh-rs/src/node.rs

Lines changed: 34 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,14 @@ pub struct NodeImpl {
1212
pub namespace: CString,
1313
pub fq_name: CString,
1414
pub graph_guard_condition: *mut rmw_guard_condition_t,
15+
/// The shared state registered with hiroz's graph-event manager.
16+
///
17+
/// Kept so teardown can unregister the exact allocation it registered
18+
/// (identity is `Arc::ptr_eq`). Holding it here also means the state
19+
/// survives `rmw_destroy_guard_condition` below, which is what makes it safe
20+
/// to free the C handle while a graph-change trigger may still be in flight.
21+
pub graph_guard_condition_state:
22+
Option<std::sync::Arc<dyn hiroz::event::GraphGuardCondition>>,
1523
}
1624

1725
impl NodeImpl {
@@ -44,6 +52,7 @@ impl NodeImpl {
4452
namespace: namespace_cstr,
4553
fq_name: fq_name_cstr,
4654
graph_guard_condition: std::ptr::null_mut(),
55+
graph_guard_condition_state: None,
4756
})
4857
}
4958
}
@@ -122,12 +131,23 @@ pub extern "C" fn rmw_create_node(
122131
}
123132
node_impl.graph_guard_condition = graph_guard_condition;
124133

125-
// Register the graph guard condition with the graph event manager
134+
// Register the graph guard condition with the graph event manager.
135+
//
136+
// Register the *shared state*, not the C pointer: the manager triggers with
137+
// its registry lock released, so a raw pointer could be freed by a
138+
// concurrent `rmw_destroy_node` between snapshot and call. Handing over an
139+
// `Arc` keeps the target alive for the duration of any in-flight trigger.
140+
let gc_state: std::sync::Arc<dyn hiroz::event::GraphGuardCondition> =
141+
match graph_guard_condition.borrow_data() {
142+
Ok(gc_impl) => gc_impl.share_state(),
143+
Err(_) => return std::ptr::null_mut(),
144+
};
145+
node_impl.graph_guard_condition_state = Some(gc_state.clone());
126146
node_impl
127147
.inner
128148
.graph()
129149
.event_manager
130-
.register_graph_guard_condition(graph_guard_condition as *mut std::ffi::c_void);
150+
.register_graph_guard_condition(gc_state);
131151

132152
// Add node to local graph for immediate discovery
133153
if let Err(e) = node_impl
@@ -175,7 +195,7 @@ pub extern "C" fn rmw_destroy_node(node: *mut rmw_node_t) -> rmw_ret_t {
175195
}
176196

177197
// Remove node from local graph and destroy the graph guard condition
178-
if let Ok(node_impl) = node.borrow_data() {
198+
if let Ok(node_impl) = node.borrow_mut_data() {
179199
// Remove node from local graph
180200
if let Err(e) = node_impl
181201
.inner
@@ -188,14 +208,17 @@ pub extern "C" fn rmw_destroy_node(node: *mut rmw_node_t) -> rmw_ret_t {
188208
}
189209

190210
if !node_impl.graph_guard_condition.is_null() {
191-
// Unregister from graph event manager
192-
node_impl
193-
.inner
194-
.graph()
195-
.event_manager
196-
.unregister_graph_guard_condition(
197-
node_impl.graph_guard_condition as *mut std::ffi::c_void,
198-
);
211+
// Unregister from graph event manager, by the same allocation we
212+
// registered. Destroying the C handle immediately afterwards is
213+
// safe even if a graph-change trigger is in flight: that trigger
214+
// holds its own `Arc` to the state, which outlives this handle.
215+
if let Some(gc_state) = node_impl.graph_guard_condition_state.take() {
216+
node_impl
217+
.inner
218+
.graph()
219+
.event_manager
220+
.unregister_graph_guard_condition(&gc_state);
221+
}
199222
crate::guard_condition::rmw_destroy_guard_condition(node_impl.graph_guard_condition);
200223
}
201224
}

0 commit comments

Comments
 (0)